Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Googlebot getting caught in robots.txt spider trap


johnnie - 1:26 am on Sep 5, 2011 (gmt 0)


It's a common misconception that robots.txt completely blocks out Googlebot. Googlebot still discovers and even indexes locations blocked by robots.txt. It just doesnt read, nor does it index the actual content of the page.

If you really want to physically block Googlebot, use an IP filter.


Thread source:: http://www.webmasterworld.com/google/4346138.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com