1 Replies - 278 Views - Last Post: 12 January 2013 - 10:50 PM Rate Topic: -----

#1 ybadragon  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 274
  • View blog
  • Posts: 1,459
  • Joined: 11-May 12

code not writing out all text in .txt

Posted 12 January 2013 - 04:08 PM

Hello,
I am having an issue with a program I've been working on that is a MTG Deck Builder. What this program specifically does is go to gatherer.wizards.com and reads the source code behind the page to get every card ever printed, store them into an arraylist, and the loop through the list writing each line to a .txt file. It always stops at Zhalfir Crusader. The last line it writes is "Shalfir Crusa" then just stops. I can't figure out why it's doing this and help would be greatly appreciated.

Attached is the program itself, just run the program and press Local Storage.

Thank you,

View Postybadragon, on 12 January 2013 - 11:05 PM, said:

Hello,
I am having an issue with a program I've been working on that is a MTG Deck Builder. What this program specifically does is go to gatherer.wizards.com and reads the source code behind the page to get every card ever printed, store them into an arraylist, and the loop through the list writing each line to a .txt file. It always stops at Zhalfir Crusader. The last line it writes is "Shalfir Crusa" then just stops. I can't figure out why it's doing this and help would be greatly appreciated.

Attached is the program itself, just run the program and press Local Storage.

Thank you,


It's not letting me attach, so here is the code
Private Sub btnLocalStorage_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnLocalStorage.Click
        Dim saveDialog As SaveFileDialog = New SaveFileDialog
        Dim dlgResult As DialogResult = New DialogResult
        Dim localStream As StreamReader
        Dim localWrite As StreamWriter
        Dim localWebReq As Net.WebRequest
        Dim localWebResp As Net.WebResponse
        Dim cdIDs As New ArrayList
        Dim cdNames As New ArrayList
        Dim count As Integer = 0
        Dim cdName As String
        Dim foundEnglish As Boolean = False
        Dim addedSets As Boolean = False
        Dim cdID As Integer

        ' ProgressBar
        'ProgressBar1.Maximum = 130
        'ProgressBar1.Show()


        Dim webReqString As String = "http://gatherer.wizards.com/Pages/Search/Default.aspx?output=compact&name=+%5b%22%5d"

        saveDialog.FileName = "MTGD DATA"
        saveDialog.Filter = "Text Files (*.txt)|*.txt"
        dlgResult = saveDialog.ShowDialog()

        If dlgResult = DialogResult.OK Then
            localWrite = File.CreateText(saveDialog.FileName)
            ' connect to website
            ' seperate all sets into arraylist

            localWebReq = Net.WebRequest.Create(webReqString)
            localWebResp = localWebReq.GetResponse()
            localStream = New StreamReader(localWebResp.GetResponseStream())

            'Get the max page number
            Using localStream
                While Not localStream.EndOfStream And Not foundMaxPage
                    currentLine = localStream.ReadLine()
                    If currentLine.Contains(" >") Then
                        Console.WriteLine(currentLine)
                        currentLine = currentLine.Remove(0, currentLine.IndexOf(" >"))
                        Console.WriteLine(currentLine)
                        currentLine = currentLine.Remove(0, currentLine.IndexOf("page=") + 5)
                        Console.WriteLine(currentLine)
                        maxPgNum = Convert.ToInt32(currentLine.Substring(0, 3))
                        foundMaxPage = True
                        Console.WriteLine(count.ToString())
                    End If
                End While
            End Using

            'loop through each page and get the name/multiverid of the card
            For i As Integer = 0 To maxPgNum
                Console.WriteLine("PAGE: " & i & "****************")
                webReqString = "http://gatherer.wizards.com/Pages/Search/Default.aspx?page=" & i & "&output=compact&name=+%5b%22%5d"
                localWebReq = Net.WebRequest.Create(webReqString)
                localWebResp = localWebReq.GetResponse()
                localStream = New StreamReader(localWebResp.GetResponseStream())

                Using localStream
                    While Not localStream.EndOfStream
                        currentLine = localStream.ReadLine()
                        If currentLine.Contains("multiverseid") And currentLine.Contains("cardTitle") Then
                            'Console.WriteLine(currentLine)
                            currentLine = currentLine.Remove(0, currentLine.LastIndexOf("=") + 1)
                            cdID = currentLine.Substring(0, currentLine.IndexOf(">") - 1)
                            'Console.WriteLine(cdID.ToString())
                            'Console.WriteLine(currentLine)
                            currentLine = currentLine.Remove(0, currentLine.IndexOf(">") + 1)
                            cdName = currentLine.Replace("</a>", "")
                            'Console.WriteLine(currentLine)
                            'localWrite.WriteLine(cdID.ToString() & "," & cdName.ToString())
                            cdIDs.Add(cdID)
                            cdNames.Add(cdName)
                            'Console.WriteLine(cdID.ToString() & ", " & cdName.ToString())
                        End If
                    End While
                End Using
                ' ProgressBar1.Value += 1
            Next
            'ProgressBar1.Value = 0
            'ProgressBar1.Maximum = cdIDs.Count + 1
            Using localWrite
                For i As Integer = 0 To cdIDs.Count - 1
                    localWrite.WriteLine(cdIDs(i).ToString() & "|" & cdNames(i).ToString())
                    Console.WriteLine("Wrote: " & cdIDs(i).ToString() & "&" & cdNames(i).ToString() & "to MTGD DATA")
                    'ProgressBar1.Value += 1
                Next
            End Using
            MessageBox.Show("Finished")
        End If
    End Sub



Is This A Good Question/Topic? 0
  • +

Replies To: code not writing out all text in .txt

#2 modi123_1  Icon User is online

  • Suitor #2
  • member icon



Reputation: 9479
  • View blog
  • Posts: 35,743
  • Joined: 12-June 08

Re: code not writing out all text in .txt

Posted 12 January 2013 - 10:50 PM

Sorry champ scraping their site is against their terms of use.

Quote

Such license is subject to these TOU and does not include:
...
(e) use of any data mining, robots or similar data gathering or extraction methods;
(f) downloading (other than the page caching) of any portion of the Sites, any Site Content, the Services or any information contained therein, except as expressly permitted on the Sites or pursuant to the Service

http://company.wizards.com/tou

As such we cannot help you here, and I will be closing the topic. Do not persist in asking for help on this subject. If you have questions feel free to shoot me a PM.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1