Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Googlebot getting caught in robots.txt spider trap


SEOMike - 5:14 pm on Sep 1, 2011 (gmt 0)


The same IP just got banned again. Have whitelisted once more, but it seems this googlebot is simply not adhering to the robots.txt file


I have seen Googlebot ignore noindex and robots.txt rules if content is heavily linked from other websites. Google picks up indicators that the content is important and it seems that if enough sites "vote" for the disallowed content Google will ignore your attempts to keep them out. There was a discussion on here a while back about this very topic but I can't find it.


Thread source:: http://www.webmasterworld.com/google/4346138.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com