Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Disallow all URLs with ' '


phranque - 11:34 am on Apr 2, 2012 (gmt 0)


i agree with g1smd and Andy Langton that your proper response is the provide a 301 or 410 as is appropriate for each requested url and not exclude those urls from being crawled.


Thread source:: http://www.webmasterworld.com/robots_txt/4433844.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com