Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Googlebot getting caught in robots.txt spider trap


nippi - 1:03 pm on Aug 1, 2011 (gmt 0)


I am backing his claim. I also got hit by it, some 4 months after setting up a spider trap which has until now been working fine.

The link to the spider trap is rel=Nofollowed, the folder is banned in robot.txt.

The spider trap works by banning by ip address, not user agent so its not caused by a faker - and of course robots.txt was setup up correctly and prior, it was in place days before the spider trap was turned on, and its run with no problems for months.

My logs show, it was the real google, from a real google ip address that ignored my robots.txt, ignored rel-nofollow and basically killed my site.


Thread source:: http://www.webmasterworld.com/google/4346138.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com