- - Google
- -- Google SEO News and Discussion
- ---- Pages are ranking, in spite of robots.txt or robots meta tags
encyclo - 10:06 am on Jun 2, 2006 (gmt 0)
|<meta name="robots" CONTENT="noindex, nofollow" /> |
guess what G ignored it completely and crawled the pages.
They have to crawl the pages to be able to see the noindex tag - so they are always going to fetch the page first. Question is, does it rank or is it indexed anything other than URL only?
My solution to stopping all indexing in a particular directory was to use mod_rewrite to send 403 Forbidden messages to any Googlebot user agent visiting the excluded directory.
Thread source: http://www.webmasterworld.com/google/34402.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com