Woz - 1:22 am on Jan 25, 2002 (gmt 0)

There was some debate some time ago as to whether some search engines would fail to index a site if there was no robots text file present. I would speculate that any decent spider would not be thus affected, however it would be better to include it to be safe.

Try using the search [searchengineworld.com] function of the site and search for "robots text file" without the quotes and you should find a number of threads discussing this very topic along with the original versions of Bretts Super robots file.

Also, check out the Robots Text Validator [searchengineworld.com].

Onya
Woz