[Duckington]

I was browsing my site's AWStats earlier..as you do, and came across this which seemed rather odd to me:



Is that normal to have that many hits from unknown robots? Because it seems really excessive to me...but then again, i've never really bothered to check these sorts of stats much, so maybe it's a common thing to have it so high?

Thanks.

This post has been edited by Duckington: 03 July 2012 - 04:15 PM

[modi123_1]

Is that in one month?! 18gb of bandwidth in one month is a bit excessive.

[Duckington]

That's just the last 4 days.

June was 212GB apparently.

This post has been edited by Duckington: 03 July 2012 - 04:07 PM

[Duckington]

I was going to look at just banning the offending IP addresses...but it appears to be bloody thousands of them...



EDIT: This forum cuts off the right hand side of my images, so right click and view the actual image to see how much they are all using up.

This post has been edited by Duckington: 03 July 2012 - 04:14 PM

[modi123_1]

Yeah.. that's bad. You might just start banning whole IP octets.. that or some intensive robot.txt files.

[no2pencil]

If they are aggressive or malicious bots, they are not going to adhere to the robots.txt file.

Not sure what os you are hosting with, but if this were my server on Unix/Linux, I would code a script to analyze usage & add ips to the firewall dynamically.

[JackOfAllTrades]

Smells like a botnet, eh?

Yay Windows!

[Duckington]

Hmm, I had a look at the actual access logs and what pages are being requested, and I think it's actually just the users using the site a lot more than I thought (seems that there's about 120 users online at any given moment, and usually about 10 requests to the server per second). It's a very interactive site with lots of things reloading and updating with AJAX a lot of the time, and it seems people are just using it a lot and leaving it running overnight or while they're not even using it..

Going to have to put in something to disable a lot of the features if they've been inactive for a while.