Problem with spiderbots scrounging for robot.txt

[Mr_N]

The spiderbots for Google, Inktomi, and a few of the other big boys have been doing their thing trying to index my site, but according to my logs they seem to be obsessing with trying to find a robot.txt validator. As such, they only seem to make the occasional deep scan, and the rest of the time are just trying again, and again, and again to find a robot.txt. Bare in mind this is all on the index page for the site.

There isn't anything on the site that I don't want the spiderbots getting their hands on, so I haven't bothered to put any robot.txt on the mainpage. I was under the impression that foregoing the robot.txt would just let the spiderbots do their thing unabated, but all the bots seem to really want to find it, and I'm getting tons of 404 errors as a result.

Anyone have an idea why this may be happening? Should I put some basic robot.txt validator on the page to make the bots happy?

[Birdman]

but all the bots seem to really want to find it

That's what they were 'trained' to do. Since the file(robots.txt) may change from time to time, they SHOULD be checking it often.

I wouldn't even worry about it.

Birdman

ps: Welcome to the forum

[Krapulator]

Put an empty robots.txt in your website root if the 404's are bothering you too much.