Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Bots using robots
Rajith - 10:24 am on Mar 12, 2005 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/567.htm
I agree with Matt Probert, Roborts are under no obligation but its good to have robot.txt in the site which give clear idea to robot of the search engine.
Brought to you by WebmasterWorld: http://www.webmasterworld.com