homepage Welcome to WebmasterWorld Guest from 54.197.183.230
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
Fast Spider only checked for my robots.txt
coffeelover

10+ Year Member



 
Msg#: 858 posted 7:41 pm on Apr 14, 2003 (gmt 0)

I was hit by the Fast webcrawler today, but it only looked at my non-existant robots.txt

Do I need a robots.txt file to not scare off some spiders?

Here is the sole log entry:
cr036r01-2.sac2.fastsearch.net - - [14/Apr/2003:14:55:30 -0400] "GET /robots.txt HTTP/1.0" 404 278 "-" "FAST-WebCrawler/3.8 (atw-crawler at fast dot no; [fast.no...]

No other entries.

Is this normal? Will it be back?

 

HitProf

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 858 posted 10:10 pm on Apr 14, 2003 (gmt 0)

Don't worry, they'll be back.

heini

WebmasterWorld Senior Member heini us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 858 posted 10:17 pm on Apr 14, 2003 (gmt 0)

Putting up a robots.txt is probably a good idea. But, as HitProf says, this is normal behaviour for Fast's bots.
I would expect the bot to come back at some point over the next weeks, to spider the main pages, i.e. the pages in the main navigation, and then, in a third take, get most or all of the pages.

HitProf

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 858 posted 11:32 am on Apr 15, 2003 (gmt 0)

It's my experience that they update pages in batches and that it takes some time between spidering and updating.

It's not uncommon to see some new pages show up and others not untill days or even weeks later.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved