Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Googlebot getting caught in robots.txt spider trap
tangor - 1:30 pm on Sep 3, 2011 (gmt 0)
If robots.txt is .htaccess passed to ALL regardless of any denies, then no bot can NOT see it.
Thread source:: http://www.webmasterworld.com/google/4346138.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com