Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Bots using robots
vincevincevince - 8:54 pm on Mar 26, 2005 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/567.htm
In many cases, the robots.txt file is an advantage for the robot. For example, keeping it out of never ending CGI loops.
Brought to you by WebmasterWorld: http://www.webmasterworld.com