Reading robots.txt or root directory, then leaving

[Bluestreak]

Hi all,

Is it normal for a search bot to read the robots.txt file and then leave without touching anything else? Ive noticed this behavior from MSIECrawler mostly, but also other bots as well. They'll come, read the robots.txt and then leave without touching anything else.

In addition, I've noticed some bots wuld come and read apparently nothing but the root directory, like this:

64.152.75.4 Url: /
Http Code : 200
Date: Jun 14 05:06:16 Http Version: HTTP/1.1
Size in Bytes: 10379
Referer: - Scooter/Altavista

Notice it didnt read any files or subdirectories, just apparently the root "/"

Just want to make sure this is normal and not indicative of a problem...

[engine]

It may just be checking the site is still live and to retain a live record in the database. I'm assuming it has indexed your site at some point previously.

[Bluestreak]

Yep, all the major bots came in and deep crawled my site to the ends of the earth in the recent past. :D