jdMorgan - 3:14 pm on Oct 5, 2002 (gmt 0)

jacon4,

The point being that you should at least consider a blank robots.txt or one which contains only:

User-agent: *
Disallow:

This will prevent filling up your server logs with a whole bunch of 404-Not Found errors as robots try to fetch robots.txt while they spider your site. And since it contains no filenames in Disallow directives, I doubt it poses a security risk.

You could also build the robots.txt without regard to security issues, and then use second-tier techniques to secure your site, such as using .htaccess or scripting to "trap" access attempts which should not have been made by any User-agent which obeys robots.txt. I use a mixture of these techniques, to good effect.

Jim