robots.txt

Good for spiders or a sitting duck?

Page 1 of 1

2 Replies - 509 Views - Last Post: 01 June 2009 - 11:21 PM

#1 no2pencil  Icon User is offline

  • Head MFIC
  • member icon

Reputation: 5063
  • View blog
  • Posts: 26,437
  • Joined: 10-May 07

robots.txt

Posted 01 June 2009 - 08:47 PM

I've been using robots.txt to not allow spiders to crawl specific portions of my sites. Pretty much what it's designed for. However, I just got to thinking, couldn't a malicious spider, assuming the file exists, grab the file, read it's contents, & specifically crawl those directories looking for goodies? I mean, isn't kind of like advertising what parts of your site contain the good stuff?

Does anyone have robots.txt defense measures in place? Or am I just being too paranoid? It's my experience that when it comes to the net, you can never be too paranoid.

Is This A Good Question/Topic? 0
  • +

Replies To: robots.txt

#2 firebolt  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 92
  • View blog
  • Posts: 5,561
  • Joined: 20-February 09

Re: robots.txt

Posted 01 June 2009 - 11:03 PM

I've been fiddling around with it, but have no real time to go through how it works. Hopefully I'll get to it soon.
Was This Post Helpful? 0
  • +
  • -

#3 BetaWar  Icon User is offline

  • #include "soul.h"
  • member icon

Reputation: 1104
  • View blog
  • Posts: 6,913
  • Joined: 07-September 06

Re: robots.txt

Posted 01 June 2009 - 11:21 PM

Don't know. I wouldn't be shocked if there are malware spiders out there already doing that type of thing. I haven't ever really done anything with Rotobs.txt, as I don't know that I have anything worth keeping off of search engines (RG is pretty sickly these days, getting more spam bots than new people, and hardly any posts).

Would be interesting to see what Robots.txt actually does for you in case of mal ware... Wonder if they ever thought of that.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1