I just had this idea.

Okay so I always make spreadsheets for building my new computer builds, and I compare prices from different websites. And I have to keep going to those websites to see if there is an update on the price or whatever or if there is a sale or something. Well what if I created a program that did all that for me. You could fill in the part name or something or add links and it would go check those links and automatically update the prices in the program and you would be able to see if there is anything going on with out having to go to all the websites....

I had a question, is it possible to retrieve information from a website using .net. I have seen other places do it like pricegrabber but im not sure if they have access to the actual database of the website because they are non profit or what.

let me know what you think

I did find this but i havnt looked into it that much yet

http://www.codeplex.com/htmlagilitypack

I also know if the website has a web service then i can grab it with .NET and parse it into and XML document but i dont think all the websites i want are going to have that... any ideas??

Generally, these people are doing something known as "screen scraping" - they don't have access to the database. They go to the page, and know how the page's source is laid out - so they know what to look for to find the data they need. Screen scraping, while an easy way to get the data, isn't a reliable way - if the page structure suddenly changes, chances are pretty high that it will break your scraper.

We use several scraping applications at work (we wrote them of course) and if it's built properly they can change the layout of the page however they want. The best way is to use Regular Expressions to search for links and item's you're after.

I think this sounds like a good tutorial for me to work on LOL!

PsychoCoder's right - if you build your scraper properly, so long as the page doesn't change too drastically(like amazon.com turning into ebay or something), you should be okay. You do still have to be careful though, as page changes can break things.

I would LOVE a tutorial on this. Wow, how interesting...

thats what i was kinda looking for lol. well im going to look into the scraping thing a bit more and let you know what i find =)

so i was looking around found this.

http://will.incorrige.us/price-checker.cgi...ortby=costperGB

can anyone tell me how this guy did it, im not that familiar with what he did...