Msg#: 3726894 posted 9:55 am on Aug 20, 2008 (gmt 0)
The robots.txt file must reside in the root of the domain and must be named "robots.txt". Bots only check for this file in the root of the domain. For instance, http://www.example.com/robots.txt is a valid location.
Msg#: 3726894 posted 9:56 am on Aug 20, 2008 (gmt 0)
You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file.