Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Why should I have a robots.txt file?


goodroi - 9:17 pm on Jan 4, 2007 (gmt 0)


If a search engine crawls a page and then you block it with robots.txt, that page will eventually fall out of the search index.

If you accidentally allowed content to be published on two sites, you need to choose one and block the other or you will have problems with the engines.

Not sure where the dilemma is.


Thread source:: http://www.webmasterworld.com/robots_txt/3203372.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com