Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Googlebot getting caught in robots.txt spider trap


Staffa - 10:41 pm on Sep 4, 2011 (gmt 0)


Gbot doesn't always read robots.txt before each crawl of one or more pages.
The occurrences I have seen on one of my sites, the bot went straight for the disallowed folder without first reading the robots.txt (which has remained unchanged for several years) and that was before the start of +1 buttons.


Thread source:: http://www.webmasterworld.com/google/4346138.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com