homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

fast spidering with libwww user agent
jeremy goodrich

 5:17 am on Oct 23, 2001 (gmt 0)

fpcr001.sac2.fastsearch.net - - [22/Oct/2001:21:39:31 -0400] "GET /mypage.html HTTP/1.0" 200 8360 "-" "libwww-perl/5.
52 FP/4.0"
fpcr001.sac2.fastsearch.net - - [22/Oct/2001:21:39:38 -0400] "GET /the-other-one.html HTTP/1.0" 200 8241 "-" "libwww-perl/5.52 FP/4.0"

What I'm wondering is does this mean that they forgot (when they were going around incognito) to check for the "if modified since" date on the pages in question, or when they are doing the 'cloak busting' thing, they just ignore that...

Just curious. It's funny to see spiders hit a site more times than they send referrals...even with good rankings.



 7:18 am on Oct 23, 2001 (gmt 0)

That could also have been a simple hand check too. I've seen it a few times. Not agressive spidering with it though. Spot checks here and there - can't say as I blame them.

We all have different experiences. I talked with a guy today that was getting 3k referrals a day out of altavista to a modest sized site (shock me...). Same is true with Fast - I get great traffic from fast via Lycos and Sympatico.


 7:54 am on Oct 23, 2001 (gmt 0)

Fast/Lycos is my third biggest referrer, behind MSN and Google.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved