5 Replies - 11849 Views - Last Post: 19 January 2010 - 11:56 AM Rate Topic: -----

#1 renoo  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 2
  • Joined: 17-January 10

Guides to create a web crawler using vb.net

Post icon  Posted 18 January 2010 - 03:31 AM

Guides to create a web crawler using vb.net?
Is This A Good Question/Topic? 0
  • +

Replies To: Guides to create a web crawler using vb.net

#2 AMDKilla  Icon User is offline

  • D.I.C Head

Reputation: 6
  • View blog
  • Posts: 88
  • Joined: 30-December 09

Re: Guides to create a web crawler using vb.net

Posted 18 January 2010 - 03:47 AM

Surely you could just index google, or use their results in some form? Would save a fair bit of bandwidth and programming on your behalf.
Was This Post Helpful? 0
  • +
  • -

#3 renoo  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 2
  • Joined: 17-January 10

Re: Guides to create a web crawler using vb.net

Posted 18 January 2010 - 06:46 AM

View PostAMDKilla, on 18 Jan, 2010 - 02:47 AM, said:

Surely you could just index google, or use their results in some form? Would save a fair bit of bandwidth and programming on your behalf.



I am just going to crawl few website such as msn/weather and yahoo/weather.I need to know how do I create a web based web crawler and extract the weather information?
Was This Post Helpful? 0
  • +
  • -

#4 Amrykid  Icon User is offline

  • 4+1=Moo
  • member icon

Reputation: 148
  • View blog
  • Posts: 1,589
  • Joined: 16-December 08

Re: Guides to create a web crawler using vb.net

Posted 18 January 2010 - 06:59 AM

View Postrenoo, on 18 Jan, 2010 - 06:46 AM, said:

View PostAMDKilla, on 18 Jan, 2010 - 02:47 AM, said:

Surely you could just index google, or use their results in some form? Would save a fair bit of bandwidth and programming on your behalf.



I am just going to crawl few website such as msn/weather and yahoo/weather.I need to know how do I create a web based web crawler and extract the weather information?

First you must master the ninja art of....*drum roll*........ Screen Scraping! That is when you get html from a website and retrieve data from that html.
The code below fetchs the html for you.
Dim wb as new WebClient
Dim html as string = wb.DownloadString("http://www.msn.com")


^^ That was the easy part. Now you just scan through the html using Regex to find what you want! If you need anymore info, hit me up through PM, otherwise, if my post was helpful, click it below! :D Good Luck!

This post has been edited by Amrykid: 18 January 2010 - 07:07 AM

Was This Post Helpful? 1
  • +
  • -

#5 AMDKilla  Icon User is offline

  • D.I.C Head

Reputation: 6
  • View blog
  • Posts: 88
  • Joined: 30-December 09

Re: Guides to create a web crawler using vb.net

Posted 18 January 2010 - 08:04 PM

Or, the best way to do it would be to find the XML (I'm guessing) file that is used for the weather gadgets. No idea where they are, but they are readily accessable by gadgets in Vista and Win 7, so you must be able to get hold of them.
Was This Post Helpful? 0
  • +
  • -

#6 Jack Eagles1  Icon User is offline

  • Pugnacious Penguin (inspired by no2pencil)
  • member icon

Reputation: 183
  • View blog
  • Posts: 1,152
  • Joined: 10-December 08

Re: Guides to create a web crawler using vb.net

Posted 19 January 2010 - 11:56 AM

You can use the HtmlElement.GetAttribute method to extract the text values of certain elements.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1