jdMorgan - 2:07 am on May 18, 2003 (gmt 0)

SinclairUser,

You get a lot of 404 errors from 'bots trying to find it, cluttering up your error logs and hiding real errors!

Other than that, the lack of a robots.txt file is interpreted by robots to mean, "request anything you like."

A good default robots.txt file which allows unlimited access but prevents all those 404s is:

User-agent: *
Disallow:

Follow the "Disallow:" line with one blank line - some obscure old robots require it.

Jim