Fast Spider only checked for my robots.txt

[coffeelover]

I was hit by the Fast webcrawler today, but it only looked at my non-existant robots.txt

Do I need a robots.txt file to not scare off some spiders?

Here is the sole log entry:
cr036r01-2.sac2.fastsearch.net - - [14/Apr/2003:14:55:30 -0400] "GET /robots.txt HTTP/1.0" 404 278 "-" "FAST-WebCrawler/3.8 (atw-crawler at fast dot no; [fast.no...]

No other entries.

Is this normal? Will it be back?

[HitProf]

Don't worry, they'll be back.

[heini]

Putting up a robots.txt is probably a good idea. But, as HitProf says, this is normal behaviour for Fast's bots.
I would expect the bot to come back at some point over the next weeks, to spider the main pages, i.e. the pages in the main navigation, and then, in a third take, get most or all of the pages.

[HitProf]

It's my experience that they update pages in batches and that it takes some time between spidering and updating.

It's not uncommon to see some new pages show up and others not untill days or even weeks later.