Welcome to WebmasterWorld Guest from 50.19.156.133

Forum Moderators: open

Message Too Old, No Replies

Fast Spider only checked for my robots.txt

     

coffeelover

7:41 pm on Apr 14, 2003 (gmt 0)

10+ Year Member



I was hit by the Fast webcrawler today, but it only looked at my non-existant robots.txt

Do I need a robots.txt file to not scare off some spiders?

Here is the sole log entry:
cr036r01-2.sac2.fastsearch.net - - [14/Apr/2003:14:55:30 -0400] "GET /robots.txt HTTP/1.0" 404 278 "-" "FAST-WebCrawler/3.8 (atw-crawler at fast dot no; [fast.no...]

No other entries.

Is this normal? Will it be back?

HitProf

10:10 pm on Apr 14, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Don't worry, they'll be back.

heini

10:17 pm on Apr 14, 2003 (gmt 0)

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Putting up a robots.txt is probably a good idea. But, as HitProf says, this is normal behaviour for Fast's bots.
I would expect the bot to come back at some point over the next weeks, to spider the main pages, i.e. the pages in the main navigation, and then, in a third take, get most or all of the pages.

HitProf

11:32 am on Apr 15, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It's my experience that they update pages in batches and that it takes some time between spidering and updating.

It's not uncommon to see some new pages show up and others not untill days or even weeks later.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month