Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Disallow all URLs with ' '


danwhitehouse - 3:41 pm on Mar 29, 2012 (gmt 0)


Hi Andy,
That's a great help with the wildcard. It seems like a very easy solution.

The problem comes from there being 1000s of redundant URLs with '_' in them. We'd doing all our redirects through .htaccess . That's be a big file if those redirects were added.


Thread source:: http://www.webmasterworld.com/robots_txt/4433844.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com