homepage Welcome to WebmasterWorld Guest from 54.234.2.94
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
Fast Spider only checked for my robots.txt
coffeelover




msg:223054
 7:41 pm on Apr 14, 2003 (gmt 0)

I was hit by the Fast webcrawler today, but it only looked at my non-existant robots.txt

Do I need a robots.txt file to not scare off some spiders?

Here is the sole log entry:
cr036r01-2.sac2.fastsearch.net - - [14/Apr/2003:14:55:30 -0400] "GET /robots.txt HTTP/1.0" 404 278 "-" "FAST-WebCrawler/3.8 (atw-crawler at fast dot no; [fast.no...]

No other entries.

Is this normal? Will it be back?

 

HitProf




msg:223055
 10:10 pm on Apr 14, 2003 (gmt 0)

Don't worry, they'll be back.

heini




msg:223056
 10:17 pm on Apr 14, 2003 (gmt 0)

Putting up a robots.txt is probably a good idea. But, as HitProf says, this is normal behaviour for Fast's bots.
I would expect the bot to come back at some point over the next weeks, to spider the main pages, i.e. the pages in the main navigation, and then, in a third take, get most or all of the pages.

HitProf




msg:223057
 11:32 am on Apr 15, 2003 (gmt 0)

It's my experience that they update pages in batches and that it takes some time between spidering and updating.

It's not uncommon to see some new pages show up and others not untill days or even weeks later.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved