Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Googlebot getting caught in robots.txt spider trap


graeme_p - 5:37 pm on Sep 6, 2011 (gmt 0)


@draftzero, that seems to imply that the page is not crawled for search purposes, which is not what the conversation above assumes. If that is really what they are doing, there is no problem.

@g1smd, part of the problem is that some CMSs get it wrong. I think Wordpress used to but it was fixed.

@lucy, on another thread you said your site was entirely static HTML, so you have nothing to worry about: I have never come across a web server getting it wrong, its badly written CMS's and scripts.


Thread source:: http://www.webmasterworld.com/google/4346138.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com