| Fast Spider only checked for my robots.txt
|
coffeelover
msg:223054 | 7:41 pm on Apr 14, 2003 (gmt 0) |
I was hit by the Fast webcrawler today, but it only looked at my non-existant robots.txt Do I need a robots.txt file to not scare off some spiders? Here is the sole log entry:
cr036r01-2.sac2.fastsearch.net - - [14/Apr/2003:14:55:30 -0400] "GET /robots.txt HTTP/1.0" 404 278 "-" "FAST-WebCrawler/3.8 (atw-crawler at fast dot no; [fast.no...] No other entries. Is this normal? Will it be back?
|
HitProf
msg:223055 | 10:10 pm on Apr 14, 2003 (gmt 0) |
Don't worry, they'll be back.
|
heini
msg:223056 | 10:17 pm on Apr 14, 2003 (gmt 0) |
Putting up a robots.txt is probably a good idea. But, as HitProf says, this is normal behaviour for Fast's bots.
I would expect the bot to come back at some point over the next weeks, to spider the main pages, i.e. the pages in the main navigation, and then, in a third take, get most or all of the pages.
|
HitProf
msg:223057 | 11:32 am on Apr 15, 2003 (gmt 0) |
It's my experience that they update pages in batches and that it takes some time between spidering and updating. It's not uncommon to see some new pages show up and others not untill days or even weeks later.
|
|
|
