Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Why should I have a robots.txt file?


pleeker - 7:58 am on Dec 30, 2006 (gmt 0)


Your impression is correct that the purpose of robots.txt is just to control what obedient bots will and won't crawl.

If the bot doesn't find a robots.txt, it assumes it's okay to crawl anything and everything it finds under that domain. If you're okay with that, then no worries. :)


Thread source:: http://www.webmasterworld.com/robots_txt/3203372.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com