Welcome to WebmasterWorld Guest from 50.16.31.61

Forum Moderators: open

Fast Spider only checked for my robots.txt

   
7:41 pm on Apr 14, 2003 (gmt 0)

10+ Year Member



I was hit by the Fast webcrawler today, but it only looked at my non-existant robots.txt

Do I need a robots.txt file to not scare off some spiders?

Here is the sole log entry:
cr036r01-2.sac2.fastsearch.net - - [14/Apr/2003:14:55:30 -0400] "GET /robots.txt HTTP/1.0" 404 278 "-" "FAST-WebCrawler/3.8 (atw-crawler at fast dot no; [fast.no...]

No other entries.

Is this normal? Will it be back?

10:10 pm on Apr 14, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Don't worry, they'll be back.
10:17 pm on Apr 14, 2003 (gmt 0)

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Putting up a robots.txt is probably a good idea. But, as HitProf says, this is normal behaviour for Fast's bots.
I would expect the bot to come back at some point over the next weeks, to spider the main pages, i.e. the pages in the main navigation, and then, in a third take, get most or all of the pages.
11:32 am on Apr 15, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It's my experience that they update pages in batches and that it takes some time between spidering and updating.

It's not uncommon to see some new pages show up and others not untill days or even weeks later.

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month