Lately I have been going through a rather large push to do three things. Index as much of the web as I can with 1 50mb/s connection, parse the content and make it searchable, and create a nice back-end for these operations. Thus far the project is going quite nicely. I had to move away from a hosting provider and start hosting the crawler from...
In 2004 I started a project on a search engine but I have thus far closed down the project due to a lack of content that I have to search through. It did crawl a page but it wouldn't index the links or check through them at a later date. Recently I started working on a web crawler that goes though a page, copy the links, store them into the...
Leave Comment
...
| 