3 Replies - 1528 Views - Last Post: 29 August 2012 - 06:57 PM

#1 dreamincodehamza  Icon User is offline

  • D.I.C Regular
  • member icon

Reputation: -12
  • View blog
  • Posts: 330
  • Joined: 12-September 08

robot.txt issues

Posted 24 August 2012 - 02:15 PM

please see me robot file
# global
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category/*/*
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /*?
Allow: /wp-content/uploads/

#And here it is my subdir

# Disallow Other subdirs
Disallow: /test/
Disallow: /livelimmitless/
Disallow: /demos/






The Issue
robots not crawlering on my site. and not indexing it .
please see the file and help me with this what excactly the issue is.

thanks

Is This A Good Question/Topic? 0
  • +

Replies To: robot.txt issues

#2 exiles.prx  Icon User is offline

  • D.I.C Head

Reputation: 65
  • View blog
  • Posts: 239
  • Joined: 22-November 10

Re: robot.txt issues

Posted 24 August 2012 - 06:51 PM

It looks like you are blocking almost everything except the index.php page and the uploads directory from getting indexed. What do you want indexed?

Also, not sure if you using clean URL's on your site (I assume you are), but if you are not, with the current robots.txt file you are blocking ALL URL's from getting indexed that contain a '?'.

This post has been edited by exiles.prx: 24 August 2012 - 07:03 PM

Was This Post Helpful? 0
  • +
  • -

#3 dreamincodehamza  Icon User is offline

  • D.I.C Regular
  • member icon

Reputation: -12
  • View blog
  • Posts: 330
  • Joined: 12-September 08

Re: robot.txt issues

Posted 29 August 2012 - 06:18 PM

View Postexiles.prx, on 24 August 2012 - 06:51 PM, said:

It looks like you are blocking almost everything except the index.php page and the uploads directory from getting indexed. What do you want indexed?

Also, not sure if you using clean URL's on your site (I assume you are), but if you are not, with the current robots.txt file you are blocking ALL URL's from getting indexed that contain a '?'.



basically i am willing to secure my site and need to have a proper wp robot.txt file so required things get indexed only.
Was This Post Helpful? 0
  • +
  • -

#4 no2pencil  Icon User is online

  • Toubabo Koomi
  • member icon

Reputation: 5224
  • View blog
  • Posts: 26,991
  • Joined: 10-May 07

Re: robot.txt issues

Posted 29 August 2012 - 06:57 PM

One thing that you must understand about robots.txt & security, is that the robots.txt file is a guide for friendly spiders & is not in any way a security measure. For security you will want to use folder permissions or .htaccess password protection.

The robots.txt file will guide a friendly spider such as Yahoo! or Google to the content that you wish to offer. If an aggressiveness spider chooses to ignore your robots.txt, there is nothing in place to prevent him from crawling the site. If it is on the web, & there are no permissions preventing a browser from viewing it, then there is nothing preventing an aggressive spider from crawling it.

Lastly, robots.txt is webhosting, not web development, so I will move it to the correct location :)
Was This Post Helpful? 1
  • +
  • -

Page 1 of 1