Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Googlebot getting caught in robots.txt spider trap


starchild - 12:58 pm on Aug 1, 2011 (gmt 0)


No, It has been in place for many months. No changes. All good. Then I got a notification it was blocked (via the spidertrap notifier).

Sure enough, it was. Upon double checking, Google webmaster tools reported a 403 forbidden error. IP was google.

I whitelisted it, and Google webmaster tools then gave a success.


Thread source:: http://www.webmasterworld.com/google/4346138.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com