Welcome to WebmasterWorld Guest from 188.8.131.52 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Subscribe to WebmasterWorld
Fast Spider only checked for my robots.txt coffeelover msg:223054 7:41 pm on Apr 14, 2003 (gmt 0) I was hit by the Fast webcrawler today, but it only looked at my non-existant robots.txt
Do I need a robots.txt file to not scare off some spiders?
Here is the sole log entry:
cr036r01-2.sac2.fastsearch.net - - [14/Apr/2003:14:55:30 -0400] "GET /robots.txt HTTP/1.0" 404 278 "-" "FAST-WebCrawler/3.8 (atw-crawler at fast dot no; [ ...] fast.no
No other entries.
Is this normal? Will it be back?
HitProf msg:223055 10:10 pm on Apr 14, 2003 (gmt 0)
Don't worry, they'll be back. heini msg:223056 10:17 pm on Apr 14, 2003 (gmt 0)
Putting up a robots.txt is probably a good idea. But, as HitProf says, this is normal behaviour for Fast's bots. I would expect the bot to come back at some point over the next weeks, to spider the main pages, i.e. the pages in the main navigation, and then, in a third take, get most or all of the pages. HitProf msg:223057 11:32 am on Apr 15, 2003 (gmt 0)
It's my experience that they update pages in batches and that it takes some time between spidering and updating.
It's not uncommon to see some new pages show up and others not untill days or even weeks later.