Error 404 caused by SE but not other Spiders?

[Arkanoid1984]

Hi,

I got the error 404 when they tried to open robots.txt
Mozilla/5.0 (Slurp/si; [e-mail];http://www.inktomi.com/slurp.html)

Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; www.mysitenameishere.com; +http://www.galaxy.com/info/crawler.html)

Mozilla/4.0 compatible ZyBorg/1.0 ([e-mail]; [WISEnutbot.com)...]

The same day I got the visit of ,ia_archiver,Scrubtheweb and Pompos those spiders didn't make an error 404. So why a few spiders make an error 404 and others don't? I validate my robots.txt the file is ok.
Opinions will be appreciate.

PS:I have 3 domains sharing the public_html directory .Should I have 3 robots.txt
My structure is like this
www.site1.com -> /public_html
www.site2.com -> /public_html/site2/
www.site3.com -> /public_html/site3/
May be I do need 1 robots.txt for each domain.I dont know

[scooch]

Are those directories the root directories for each respective URL?

If so, then you definitely need a robots.txt in each root directory.

Sounds to me like you're collecting logs for all three domains in one log file which can get confusing. If you have access to your webserver configuration, you may want to consider setting each virtual domain to log to a different log file.