Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- GWT, Robots.txt Update Timing


phranque - 1:07 am on Jul 29, 2010 (gmt 0)


using robot.txt will exclude the directory from being recrawled but won't prevent the url(s) from being indexed or necessarily removed from the index.
you could however use X-Robots-Tag [webmasterworld.com] which wouldn't get migrated with the content unless you also move the .htaccess file.


Thread source:: http://www.webmasterworld.com/robots_txt/4176741.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com