Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Googlebot getting caught in robots.txt spider trap


leadegroot - 12:32 pm on Sep 3, 2011 (gmt 0)


Part of my daily surf is to check no pages in my various blocked sections have been indexed by google (its a site: search)
Every few months (I'd go so far as to say once or twice a year) there is a result there and its a quick trip to WMT to get it removed.
I had been assuming it is Google's bad, but nippi's explanation of a glitch in robots.txt reading would cover it, although I would have expected a bigger tranche than the 1 page I see to occur, really...
But - it definitely does happen, on old established sites where I haven't touched the robots file in years.


Thread source:: http://www.webmasterworld.com/google/4346138.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com