6 Replies - 886 Views - Last Post: 23 July 2013 - 02:32 PM

#1 vwvw  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 3
  • Joined: 23-July 13

Method of establishing new links added to directory

Posted 23 July 2013 - 09:14 AM

Hi

My aim is to create an automated system which finds and records new pages that have been added to specific area/directory of a website.

For example, if I wanted to twice daily check for new pages added to the directory of www.bearsite.com/bears/brownbears and collect information on those pages i.e. the links etc.

I am an absolute novice. I've done some coding in html and css but other than that nothing. So I don't even understand the foundations of how this would be done.

I would be really grateful if someone could provide and answer and maybe even point me in the right direction so I can study to start learning more about this.

Thanks for your time!

Vic

Is This A Good Question/Topic? 0
  • +

Replies To: Method of establishing new links added to directory

#2 modi123_1  Icon User is online

  • Suitor #2
  • member icon



Reputation: 8374
  • View blog
  • Posts: 31,122
  • Joined: 12-June 08

Re: Method of establishing new links added to directory

Posted 23 July 2013 - 09:19 AM

It depends on the page I guess.. what sort are you dealing with?

Side question - *WHY* are you scraping pages for data as they pop up? Seems a bit.. odd.
Was This Post Helpful? 0
  • +
  • -

#3 vwvw  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 3
  • Joined: 23-July 13

Re: Method of establishing new links added to directory

Posted 23 July 2013 - 10:02 AM

View Postmodi123_1, on 23 July 2013 - 09:19 AM, said:

It depends on the page I guess.. what sort are you dealing with?

Side question - *WHY* are you scraping pages for data as they pop up? Seems a bit.. odd.


Hello

Thanks for your reply.

It's for compiling a list of new products which have been added to websites.

Thanks

Vic
Was This Post Helpful? 0
  • +
  • -

#4 modi123_1  Icon User is online

  • Suitor #2
  • member icon



Reputation: 8374
  • View blog
  • Posts: 31,122
  • Joined: 12-June 08

Re: Method of establishing new links added to directory

Posted 23 July 2013 - 10:03 AM

As a side thought - why not check to see if these sites have a webservice that exposes their catalog as a readonly interface? It would be much much simpler than trying to scrape data from a site.
Was This Post Helpful? 0
  • +
  • -

#5 vwvw  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 3
  • Joined: 23-July 13

Re: Method of establishing new links added to directory

Posted 23 July 2013 - 10:14 AM

View Postmodi123_1, on 23 July 2013 - 10:03 AM, said:

As a side thought - why not check to see if these sites have a webservice that exposes their catalog as a readonly interface? It would be much much simpler than trying to scrape data from a site.


Thanks for the suggestion. I have no idea how I would check this though! Can you offer any guidance? Thank you
Was This Post Helpful? 0
  • +
  • -

#6 modi123_1  Icon User is online

  • Suitor #2
  • member icon



Reputation: 8374
  • View blog
  • Posts: 31,122
  • Joined: 12-June 08

Re: Method of establishing new links added to directory

Posted 23 July 2013 - 10:15 AM

you would contact the sites, look through their faq, or do a quick search for the company name + developer or web service.
Was This Post Helpful? 0
  • +
  • -

#7 cfoley  Icon User is offline

  • Cabbage
  • member icon

Reputation: 1906
  • View blog
  • Posts: 3,953
  • Joined: 11-December 07

Re: Method of establishing new links added to directory

Posted 23 July 2013 - 02:32 PM

This bears striking similarity to today's daily WTF:
http://thedailywtf.c...-or-3-or-4.aspx

Sounds like what you want to write is a scraper. Maybe you could start googling for techniques that would help in your chosen language.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1