School Assignment? Project Due Tomorrow? Chat LIVE With A Programming Expert!

Dream.In.Code > Web Development > Web Servers & Hosting

Welcome to Dream.In.Code
	Become an Expert! Join 300,468 Programmers for FREE! Get instant access to thousands of experts, tutorials, code snippets, and more! There are 1,711 people online right now. Registration is fast and FREE... Join Now! Chat LIVE With a Expert

robots.txt

robots.txt, Good for spiders or a sitting duck?

no2pencil	1 Jun, 2009 - 07:47 PM Post #1
i R L33t Skiddie, k? Joined: 10 May, 2007 Posts: 13,233 Thanked: 289 times Dream Kudos: 2875 Expert In: Goofing Off My Contributions	I've been using robots.txt to not allow spiders to crawl specific portions of my sites. Pretty much what it's designed for. However, I just got to thinking, couldn't a malicious spider, assuming the file exists, grab the file, read it's contents, & specifically crawl those directories looking for goodies? I mean, isn't kind of like advertising what parts of your site contain the good stuff? Does anyone have robots.txt defense measures in place? Or am I just being too paranoid? It's my experience that when it comes to the net, you can never be too paranoid.
	This Post Was Helpful!

Register for Free to Make These Ads Go Away!

firebolt	RE: Robots.txt 1 Jun, 2009 - 10:03 PM Post #2
D.I.C Lover Joined: 20 Feb, 2009 Posts: 5,463 Thanked: 75 times Dream Kudos: 1675 My Contributions	I've been fiddling around with it, but have no real time to go through how it works. Hopefully I'll get to it soon.
	This Post Was Helpful!

BetaWar	RE: Robots.txt 1 Jun, 2009 - 10:21 PM Post #3
#include <soul.h> Joined: 7 Sep, 2006 Posts: 4,729 Thanked: 269 times Dream Kudos: 1400 My Contributions	Don't know. I wouldn't be shocked if there are malware spiders out there already doing that type of thing. I haven't ever really done anything with Rotobs.txt, as I don't know that I have anything worth keeping off of search engines (RG is pretty sickly these days, getting more spam bots than new people, and hardly any posts). Would be interesting to see what Robots.txt actually does for you in case of mal ware... Wonder if they ever thought of that.
	This Post Was Helpful!

Fast Reply

Reply to this topic

Start new topic

Display Mode: Standard · Switch to: Linear+ · Switch to: Outline

Track this topic · Email this topic · Print this topic · Subscribe to this forum

Time is now: 11/8/09 02:39AM

Invision Power Board v2.1.7 © 2009 IPS, Inc.

Licensed to: MediaGroup1

Advertising | Terms of Use | Privacy Policy | About Us | Site Map

Advertise on this site through Lake Quincy Media

Forum Index: Programming Help | C and C++ | VB6 | Java | VB.NET | C# | ASP.NET | PHP | ColdFusion | Perl and Python | Ruby | Databases | Other Languages | Game Programming | Mobile Development | Software Development | Computer Science | Industry News | Programming Tutorials | C++ Tutorials | Visual Basic Tutorials | Java Tutorials | VB.NET Tutorials | C# Tutorials | Linux Tutorials | PHP Tutorials | ColdFusion Tutorials | Windows Tutorials | HTML/JavaScript Tutorials | CSS Tutorials | Flash Tutorials | Web Promotion Tutorials | Graphic Design & Photoshop Tutorials | Software Development Tutorials | Database Tutorials | Other Language Tutorials | ASP.NET Tutorials | Game Programming Tutorials | WPF & Silverlight Tutorials |

Copyright 2001-2008 MediaGroup1 LLC, All Rights Reserved
A MediaGroup1 LLC Production - Version 6.0.2.1.36
Server: secure2

Search^Updated!w00t

Over 509,000 Pages! Advanced Forum Search

Live Help!

Be Social

Tutorials

Programming

Web Development

Reference Sheets

Code Snippets

DIC Chatroom

Join our IRC Chat

Bye Bye Ads

Monthly Drawing

Top Contributors

Top 10 Kudos This Month