Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Pages are ranking, in spite of robots.txt or robots meta tags


pageoneresults - 2:43 pm on May 20, 2006 (gmt 0)


Pages that are banned in robots.txt or have noindex, nofollow on them are being indexed and are ranking poorly.

How are you determining this? Are you doing site: searches? If so, this has no bearing on anything really. If the pages are showing up for search queries, then it may be another story.

Googlebot is going to fetch everything that has a reference to it. In the instance of stuff blocked via robots.txt protocol, you're going to see a URI only listing when doing site: searches.

Want to keep those pages out of the index? Don't use the robots.txt protocol and drop a robots meta tag on that page. I've been testing this for a couple of years now and it works like a charm.


Thread source:: http://www.webmasterworld.com/google/34402.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com