Help! What just happened

I just checked my web logs. What does this mean ?

[Big_Balou]

I am definetely a newbie. Hello to all. My question is this: I just checked my web logs and both slurp from Inktomi and the Ask Jeeves spiders have come thru requesting my robot.txt file which I did not have on the page :-(. This of course returned a 404. Does this mean they didn't look any further than requesting that file ? What just happened ? Any help would be appreciated.

[bateman_ap]

I wouldn't worry. All that has happened is the spider has looked for robot.txt which is basically a file that you can create if you don't want a spider to go through certain pages or stop individual spiders completely. Do a search on the web for "writing robot.txt". As you haven't got a robot.txt file it presumes it can spider your whole site. It will prob come back in the future to spider it all.

[RussellC]

No worries, it just means they will not be disallowed. robots.txt is there to disallow certain robots to your site, or disallow robots to certain parts of your site. If you have no robots.txt they will come right in.

[Big_Balou]

Thanks a lot. I started reading one of the other threads and just wrote a robot.txt file which I will be uploading shortly. A little panic never hurt anyone I guess :->