Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Disallow all URLs with ' '


Andy_Langton - 3:34 pm on Mar 29, 2012 (gmt 0)


There shouldn't be a need to robots exclude redundant pages - if they have moved elsewhere, 301 them. If they don't exist any more, they will die out on there own.

Robots excluding them implies that they are present in search results - if they are, they should be good candidates for 301 redirecting.

If the desire is to remove the pages from search results, then a simple 404 (or 410) response is the quickest way to do that - robots exclusion won't speed this up.

All that said, if robots exclusion is the right choice in this situation, you could use a wildcard:

User-agent: googlebot
Disallow: *_*

Note that not all spiders support the wildcard - YMMV.


Thread source:: http://www.webmasterworld.com/robots_txt/4433844.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com