Hey again DreamInCode.

I have a slight problem. I've been given the challenge to make an application retrieve some data from a couple of web sites and throw it together in a .txt file. Does anyone have a good method to retrieve a web page's source code and store it as a string or similar so I can scan through it for the information I need?
I've been looking at the browser ActiveX control and scanning the web for tutorials or posts on this matter but so far without any luck.

Thanks in forehand
Sigurd

Google for "scraper C#"

Super simple my friend. You can use the System.Net.WebClient class. For example, to get the source of this topic on Dream In Code:

            System.Net.WebClient client = new System.Net.WebClient();
            client.BaseAddress = "http://www.dreamincode.net";

That will establish a connection to Dream In Code. Then, use this method

            client.DownloadString("http://www.dreamincode.net/forums/showtopic63212.htm");

That will return a string with the source of "http://www.dreamincode.net/forums/showtopic63212.htm", which happens to be this topic. Also, there is a similar method in the same class called DownloadFile(). It will download the data off the internet and automatically write it into a file. I used Dream In Code as an example, you can do this with any site.

This post has been edited by jacobjordan: 9 Sep, 2008 - 01:17 PM

A slightly more efficient approach is to use the WebRequest Class along with the StreamReader Class to accomplish this.

public string GetURLData()
{
    try
    {
        //create a new WebRequest object
        System.Net.WebRequest request = System.Net.WebRequest.Create("http://www.dreamincode.net/forums/showtopic63212.htm");

        //create StreamReader to hold the returned request
        System.IO.StreamReader stream = new System.IO.StreamReader(request.GetResponse().GetResponseStream());

        //StringBuilder to hold info from the request
        System.Text.StringBuilder builder = new System.Text.StringBuilder();

        //now loop through the response
        while (!(stream.Peek() == 0))
        {
            //now make sure we're not looking at a blank line
            if(stream.ReadLine().Length>0) builder.Append(stream.ReadLine());
        }

        //close up the StreamReader
        stream.Close();

        //return the information
        return builder.ToString();
    }
    catch (Exception ex)
    {
        //put your error handling here
        return string.Empty;
    }
}

Wow. Thanks a lot for the great responses. I think this should get the job done.

Sigurd

The presented solutions seem to work all fine but another problem has come up. The site I'm acessing is a site requiring login. Even though the user is logged in in a browser the C# application always just gets the source code of the login screen. Is there a workaround to this? Possibly allowing the user to log in using the application or something even easier?

// Edit:
I've been working around with some web page source code now and I can see that the login form calls another page called login_exec.asp. Of course some ID and Password are submitted. Don't know if this helps.

This post has been edited by SigurdSuhm: 10 Sep, 2008 - 07:54 AM