Brett_Tabke - 8:30 am on Jun 10, 2002 (gmt 0)

I realize I don't have a robots.txt on my site. Is it important to have that file in the root directory?

As others have mentioned, no. A robots.txt is for the blocking of robots that obey the standard.

Some leave it intentionally missing so that it will go 404 and show in error logs. That way, you can identify obeying spiders easy enough.

Although this is common and acceptable to standard:

User-agent: *
Disallow:

I wouldn't recommend it. There are some spiders that will incorrectly interpret that as blocking all content.

Ya jady, block anything you think is sensitive. I think cgi-bin and java would qualify for that.