[dreamincodehamza]

please see me robot file

# global
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category/*/*
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /*?
Allow: /wp-content/uploads/

#And here it is my subdir

# Disallow Other subdirs
Disallow: /test/
Disallow: /livelimmitless/
Disallow: /demos/

The Issue
robots not crawlering on my site. and not indexing it .
please see the file and help me with this what excactly the issue is.

thanks

[exiles.prx]

It looks like you are blocking almost everything except the index.php page and the uploads directory from getting indexed. What do you want indexed?

Also, not sure if you using clean URL's on your site (I assume you are), but if you are not, with the current robots.txt file you are blocking ALL URL's from getting indexed that contain a '?'.

This post has been edited by exiles.prx: 24 August 2012 - 07:03 PM

[dreamincodehamza]

exiles.prx, on 24 August 2012 - 06:51 PM, said:

It looks like you are blocking almost everything except the index.php page and the uploads directory from getting indexed. What do you want indexed?

Also, not sure if you using clean URL's on your site (I assume you are), but if you are not, with the current robots.txt file you are blocking ALL URL's from getting indexed that contain a '?'.

basically i am willing to secure my site and need to have a proper wp robot.txt file so required things get indexed only.

[no2pencil]

One thing that you must understand about robots.txt & security, is that the robots.txt file is a guide for friendly spiders & is not in any way a security measure. For security you will want to use folder permissions or .htaccess password protection.

The robots.txt file will guide a friendly spider such as Yahoo! or Google to the content that you wish to offer. If an aggressiveness spider chooses to ignore your robots.txt, there is nothing in place to prevent him from crawling the site. If it is on the web, & there are no permissions preventing a browser from viewing it, then there is nothing preventing an aggressive spider from crawling it.

Lastly, robots.txt is webhosting, not web development, so I will move it to the correct location