Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Googlebot possibly ignoring robots.txt


goodroi - 7:04 pm on Apr 26, 2012 (gmt 0)


First, I just want to mention a friendly reminder that robots.txt is a voluntary protocol. If you want to really block access to a project you should think about using .htaccess file.

Also you might want to double check that there are no typos in your existing robots.txt file. Most of the previous times that other members have reported Google not honoring robots.txt, it turned out to not be Googlebot's error.


Thread source:: http://www.webmasterworld.com/robots_txt/4446154.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com