Hi
I'm trying to choose a language to program a focused web crawler in. The purpose of this project is, more than anything, to serve as a learning experience, so things like memory usage and speed are not priorities for this crawler. I also realize that there are some open source crawlers out there, but again, I'm doing this for the learning experience. I'd like help choosing a language based on the following criteria:

-good built in functions or libraries for parsing html and xml. I'm still a relatively novice programmer, so if these can save me some time, it would be helpful.
-good support for the following character encodings: UTF-8, Shift-JIS/x-sjis, EUC-JP

Also, I already have some experience using Java, Python, and PHP, so if any of these languages fits into the criteria mentioned above, that language would be preferred.

From experience, PHP doesn't have very good support of unicode (yet), and it doesn't seem like a very well suited language for programming a web crawler in either.

While I like Python a lot, and I've heard good things about lxml and BeautifulSoup, I'm not too sure about it's unicode support after reading some of the comments here:
http://lowkster.blogspot.com/2008/06/pytho...code-sucks.html

From what I can tell, a lot of web crawlers are written in Java or C/C++. Any recommendations?

This post has been edited by chu: 21 Sep, 2008 - 11:37 AM

I think Java would fit it well. It seems to fit your criteria well and Java's a fairly simple and straight forward language.

Does anyone else think this would be good in any other languages?

Here's an article about writing a web crawler in Java if you'd like to try it out: http://java.sun.com/developer/technicalArt...rty/WebCrawler/

Hey, thanks for the replies. I guess I'll try programming the crawler in Java.

If you did do it in Java you could do it like this:
http://www.java-tips.org/java-se-tips/java...-in-java-2.html

You say you want to do this for a learning exercize... I would recommend trying a port if you really want to learn.

There are a heap of crawlers available that download text, and a few less that do images as well. If you really want to get into the nuts and bolts of crawling, and being able to reconstruct web pages I would suggest checking out http://arachnode.net.

arachnode.net is a complete C# web crawler but is straightforward enough to port to Java.

Learning to extend nutch might also be a good suggestion.