Hey,

I'm quite new to Programming and OO programming especially. Nonetheless, I'm trying to write a very simple Spider for web crawling. Here's the code:



The problem is that I can't seem to figure out how to take the @links array and put back into $URL. Like it is now, it will fetch the links and print them, but it won't follow them. Any ideas as to how I can get it to follow the links it finds?

Thanks for your help guys

$URL is your problem... It's a scalar value (indicated by the leading "$") and scalars can only hold one thing at a time. You can't put @links into $URL because @links is a list of values, not a single value.

Try this instead:

(Untested code, but the changes are simple enough that it should work at least as well as the code you posted.)

Also, you might want to take a look at WWW::Mechanize, which has a lot of this kind of functionality built into it already, as well as some other things you forgot and I didn't bother to add, such as checking robots.txt. CPAN searches for things like "spider", "crawler", or "robot" may also turn up modules which will help to minimize the amount of new code you need to write.

Thanks for that. I fixed some declarations that were needed because of strict. It crawls for the given 60 seconds, but it doesn't seem to get any links. The file that it prints to tells me "400 URL missing" . I tried changing the initial site, but that wasn't the problem. Can you see any place where this problem might be caused?

Perl variable names are case-sensitive. The first of these two lines grabs the first item from @urls and puts it into the variable $url (lower-case). The second then creates a brand-new variable named $URL (upper-case) and attempts to retrieve it; since this variable hasn't been initialized, it has no value, which is why you get the "400 URL missing" error.

Once I changed the second line toI was able to successfully retrieve http://www.youtube.com/ and 232 links were found.

The output isn't working in your code because
creates a new (and empty) @links array rather than using the existing one, so there's nothing for the loop to print. Once I removed that "my" the 232 links were printed.

Always remember that whenever you use the "my" keyword, you are creating a new variable. If you're trying to re-use an existing variable and strict complains about it, then you've probably misspelled the variable's name (either that or it's gone out of scope), which is exactly the sort of thing that strict is intended to help guard against.

A couple other side notes:

- The reason I changed your original $URL variable to $url is that Perl convention is to use all-lowercase variable names. While you can use uppercase in variable names and perl won't complain, other Perl programmers are likely to look at you funny if you ever end up sharing code with them or writing Perl in a team environment.

- Looking over the list of the 232 links found on youtube's front page, I noticed another detail you'll need to address if you want to build your own spider: Duplicate tracking. Keep a list of URLs that you've already visited (I recommend using a hash for this) and skip them if you run across them again. I can add this to your code on request to show you how to do it, but I'll give you a chance to try to work it out for yourself first.

Thank you very much. It works perfectly. Instead of your suggestion to make sure it skips links it crawled before, I was going to limit to a specific domain. This is because I plan to use it to check links on my site. I'm not 100% sure how to do that, but I would think I'm going to need some "if, else"s. One thing I don't have a clue about is how to insert the domain. Nonetheless, I will do some experiments and see what I can do. (But if what I said is completely wrong, do tell me )

For now, though, thanks so much for your help. It's very appreciated

Ok, I suddenly got this beautiful idea to disregard your tip and use an array instead. Mostly because I have no clue how to do it with a hash.

Here's the main part of the code now:


As you can see, the point of it is to push the link it just visited to the new array called "@visited". Then when it comes back around, it checks to see if "$url" and "@visited" matches, and if they do, it is supposed to skip to the next part.

Unfortunately, my sense of logic and Perl don't always get along, and this is the case here. The link only returns an "error 400". I'd appreciate it if someone could look it over and tell me where/why it's messing up.

Thanks a lot

The problem with using an array here (aside from your implementation of it not working ) is that checking an array to see whether it contains a value requires you to check the first value in the array to see if it matches, then check the second for a match, then the third, and so on. You can use the "grep" command to automate this, but, if you've got more than a handful of items in the array, it gets a bit slow. With a hash, you can determine whether a value is there or not just as quickly, no matter how many or how few values it contains.

The main advantage of arrays over hashes is that arrays maintain the order of their contents, while hashes will rearrange them into a random order. (Not really random, actually, but unpredictable enough that you may as well treat it as random for most purposes.) So use arrays when you care about the order of things and hashes when you only care whether it's there or not.



...except that's not what it checks.

has two problems:

1) "Context" is a big deal in Perl and, because $url is a scalar, that expression is evaluated in "scalar context" - basically, $url only holds a single value, so @visited gets turned into a single value so they can interact with each other. When you evaluate an array in scalar context like that, you get the number of items in the array. e.g., If @visited contains ('foo', 'bar', 'baz'), then @visited will become the value 3 in scalar context.

2) "=" is the assignment operator, so it's not comparing $url and @visited at all, it's getting the number of items in @visited and assigning that value to $url. With the sample array from the last paragraph, this would be equivalent to "if ($url = 3)". The end result is that this expression will be true (causing the program to die) if there's anything in @visited and false (causing $url to be set to 0, which leads to your "error 400") if @visited is empty. You need to use "eq" to compare strings with each other: "if ($url eq $other_url)"

So... First, here's a working way to determine whether $url is in @visited:The grep function will assign each element of the target list (@visited) to $_ in turn and return those elements where "$_ eq $url" is true. But, like I mentioned above, this will get slower and slower as @visited grows, since it needs to check each value in the array individually.

Now that you know how to do that, here's how I would do the duplicate check using a hash:

Once again I can only say thank you for your help. I'm realizing that the project might have been a little over my head, but then again, that's how you learn.

Again, Thank you so much. You really helped me a lot with this.

No problem. Helping people with Perl problems is why I come to this site.


As it is written in The Cult of Done Manifesto,

Welcome to the world of Perl! I hope you have as much fun with it as I have.