7 Replies - 2830 Views - Last Post: 04 July 2012 - 04:36 AM

#1 Duckington  Icon User is offline

  • D.I.C Addict

Reputation: 164
  • View blog
  • Posts: 599
  • Joined: 12-October 09

Unknown robots (200GB of bandwidth in 1 month)

Posted 03 July 2012 - 01:39 PM

I was browsing my site's AWStats earlier..as you do, and came across this which seemed rather odd to me:

Posted Image

Is that normal to have that many hits from unknown robots? Because it seems really excessive to me...but then again, i've never really bothered to check these sorts of stats much, so maybe it's a common thing to have it so high?

Thanks.

This post has been edited by Duckington: 03 July 2012 - 04:15 PM


Is This A Good Question/Topic? 0
  • +

Replies To: Unknown robots (200GB of bandwidth in 1 month)

#2 modi123_1  Icon User is online

  • Suitor #2
  • member icon



Reputation: 8374
  • View blog
  • Posts: 31,124
  • Joined: 12-June 08

Re: Unknown robots (200GB of bandwidth in 1 month)

Posted 03 July 2012 - 02:29 PM

Is that in one month?! 18gb of bandwidth in one month is a bit excessive.
Was This Post Helpful? 0
  • +
  • -

#3 Duckington  Icon User is offline

  • D.I.C Addict

Reputation: 164
  • View blog
  • Posts: 599
  • Joined: 12-October 09

Re: Unknown robots (200GB of bandwidth in 1 month)

Posted 03 July 2012 - 04:06 PM

That's just the last 4 days.

June was 212GB apparently.

Posted Image

This post has been edited by Duckington: 03 July 2012 - 04:07 PM

Was This Post Helpful? 0
  • +
  • -

#4 Duckington  Icon User is offline

  • D.I.C Addict

Reputation: 164
  • View blog
  • Posts: 599
  • Joined: 12-October 09

Re: Unknown robots (200GB of bandwidth in 1 month)

Posted 03 July 2012 - 04:13 PM

I was going to look at just banning the offending IP addresses...but it appears to be bloody thousands of them...

Attached Image

EDIT: This forum cuts off the right hand side of my images, so right click and view the actual image to see how much they are all using up.

This post has been edited by Duckington: 03 July 2012 - 04:14 PM

Was This Post Helpful? 0
  • +
  • -

#5 modi123_1  Icon User is online

  • Suitor #2
  • member icon



Reputation: 8374
  • View blog
  • Posts: 31,124
  • Joined: 12-June 08

Re: Unknown robots (200GB of bandwidth in 1 month)

Posted 04 July 2012 - 12:40 AM

Yeah.. that's bad. You might just start banning whole IP octets.. that or some intensive robot.txt files.
Was This Post Helpful? 0
  • +
  • -

#6 no2pencil  Icon User is online

  • Head MFIC
  • member icon

Reputation: 5066
  • View blog
  • Posts: 26,448
  • Joined: 10-May 07

Re: Unknown robots (200GB of bandwidth in 1 month)

Posted 04 July 2012 - 12:48 AM

If they are aggressive or malicious bots, they are not going to adhere to the robots.txt file.

Not sure what os you are hosting with, but if this were my server on Unix/Linux, I would code a script to analyze usage & add ips to the firewall dynamically.
Was This Post Helpful? 0
  • +
  • -

#7 JackOfAllTrades  Icon User is offline

  • Saucy!
  • member icon

Reputation: 5954
  • View blog
  • Posts: 23,219
  • Joined: 23-August 08

Re: Unknown robots (200GB of bandwidth in 1 month)

Posted 04 July 2012 - 04:22 AM

Smells like a botnet, eh?

Yay Windows!
Was This Post Helpful? 0
  • +
  • -

#8 Duckington  Icon User is offline

  • D.I.C Addict

Reputation: 164
  • View blog
  • Posts: 599
  • Joined: 12-October 09

Re: Unknown robots (200GB of bandwidth in 1 month)

Posted 04 July 2012 - 04:36 AM

Hmm, I had a look at the actual access logs and what pages are being requested, and I think it's actually just the users using the site a lot more than I thought (seems that there's about 120 users online at any given moment, and usually about 10 requests to the server per second). It's a very interactive site with lots of things reloading and updating with AJAX a lot of the time, and it seems people are just using it a lot and leaving it running overnight or while they're not even using it..

Going to have to put in something to disable a lot of the features if they've been inactive for a while.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1