Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Googlebot getting caught in robots.txt spider trap


starchild - 1:01 pm on Aug 1, 2011 (gmt 0)


BTW, the spidertrap i use to prevent crawlers using bandwidth. The site is pretty big with 100,000's of pages.

It isn't there for security reason.


Thread source:: http://www.webmasterworld.com/google/4346138.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com