how to crawl each web page of a search engine.

I want to be able to crawl each page of a search query and store the i

Page 1 of 1

2 Replies - 3127 Views - Last Post: 25 August 2008 - 08:03 PM Rate Topic: -----

#1 EvolutionMedia  Icon User is offline

  • D.I.C Head

Reputation: 3
  • View blog
  • Posts: 125
  • Joined: 11-August 08

how to crawl each web page of a search engine.

Post icon  Posted 25 August 2008 - 10:48 AM

Hey

I want to be able to start at page one of a search query and get all the images from the site then move on to the next page and collect all the data then the next.

Here's the crawl routine:
Sub CrawlPages(ByVal max_pages As Short, ByVal url As String, ByVal type As String)
		Dim i As Short
		Dim page_data As StringCollection

		Select Case type
			Case "Image"
				System.Windows.Forms.Application.DoEvents()
				For Each i As Integer In page_data
			 ' I don't know what to put here and plus I don't think I have it right.

				Next
		End Select

	End Sub



It says that 'i' has a hidden varible inside the block. So basically this routine will do is crawl to the maxium pages the user specified in the new search form then on each page it will obtain data storing it into a string collection. After it's done then it will prepare it to download into the cache folder.

Thanks guys!

Is This A Good Question/Topic? 0
  • +

Replies To: how to crawl each web page of a search engine.

#2 Jayman  Icon User is offline

  • Student of Life
  • member icon

Reputation: 418
  • View blog
  • Posts: 9,532
  • Joined: 26-December 05

Re: how to crawl each web page of a search engine.

Posted 25 August 2008 - 07:04 PM

You have declared 'i' twice in your code, once at the top the procedure and a second time in the For Each loop. Rename one of the two declarations to something different.

For example:
Sub CrawlPages(ByVal max_pages As Short, ByVal url As String, ByVal type As String)
		Dim i As Short
		Dim page_data As StringCollection

		Select Case type
			Case "Image"
				System.Windows.Forms.Application.DoEvents()
				For Each item As Integer In page_data
			 ' I don't know what to put here and plus I don't think I have it right.

				Next item
		End Select

	End Sub

Was This Post Helpful? 0
  • +
  • -

#3 gbertoli3  Icon User is offline

  • DIC at Heart + Code
  • member icon

Reputation: 40
  • View blog
  • Posts: 1,162
  • Joined: 23-June 08

Re: how to crawl each web page of a search engine.

Posted 25 August 2008 - 08:03 PM

Google Search: http://www.google.co...=...net&spell=1

or try

http://www.vbdotneth...oadWebPage.aspx

Hope this helps
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1