homepage Welcome to WebmasterWorld Guest from 54.196.24.103
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Accredited PayPal World Seller

Visit PubCon.com
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
fast spidering with libwww user agent
jeremy goodrich




msg:223519
 5:17 am on Oct 23, 2001 (gmt 0)

fpcr001.sac2.fastsearch.net - - [22/Oct/2001:21:39:31 -0400] "GET /mypage.html HTTP/1.0" 200 8360 "-" "libwww-perl/5.
52 FP/4.0"
fpcr001.sac2.fastsearch.net - - [22/Oct/2001:21:39:38 -0400] "GET /the-other-one.html HTTP/1.0" 200 8241 "-" "libwww-perl/5.52 FP/4.0"

What I'm wondering is does this mean that they forgot (when they were going around incognito) to check for the "if modified since" date on the pages in question, or when they are doing the 'cloak busting' thing, they just ignore that...

Just curious. It's funny to see spiders hit a site more times than they send referrals...even with good rankings.

 

Brett_Tabke




msg:223520
 7:18 am on Oct 23, 2001 (gmt 0)

That could also have been a simple hand check too. I've seen it a few times. Not agressive spidering with it though. Spot checks here and there - can't say as I blame them.

We all have different experiences. I talked with a guy today that was getting 3k referrals a day out of altavista to a modest sized site (shock me...). Same is true with Fast - I get great traffic from fast via Lycos and Sympatico.

IanTurner




msg:223521
 7:54 am on Oct 23, 2001 (gmt 0)

Fast/Lycos is my third biggest referrer, behind MSN and Google.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved