Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Googlebot getting caught in robots.txt spider trap


starchild - 9:25 am on Aug 30, 2011 (gmt 0)


Well,

The same IP just got banned again. Have whitelisted once more, but it seems this googlebot is simply not adhering to the robots.txt file

Confirmed at webmastertools, and indeed it was caught.

Makes it very hard when google doesn't follow it;s own directives,


Thread source:: http://www.webmasterworld.com/google/4346138.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com