homepage Welcome to WebmasterWorld Guest from 54.166.39.179
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
fast spidering with libwww user agent
jeremy goodrich

WebmasterWorld Senior Member jeremy_goodrich us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 337 posted 5:17 am on Oct 23, 2001 (gmt 0)

fpcr001.sac2.fastsearch.net - - [22/Oct/2001:21:39:31 -0400] "GET /mypage.html HTTP/1.0" 200 8360 "-" "libwww-perl/5.
52 FP/4.0"
fpcr001.sac2.fastsearch.net - - [22/Oct/2001:21:39:38 -0400] "GET /the-other-one.html HTTP/1.0" 200 8241 "-" "libwww-perl/5.52 FP/4.0"

What I'm wondering is does this mean that they forgot (when they were going around incognito) to check for the "if modified since" date on the pages in question, or when they are doing the 'cloak busting' thing, they just ignore that...

Just curious. It's funny to see spiders hit a site more times than they send referrals...even with good rankings.

 

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 337 posted 7:18 am on Oct 23, 2001 (gmt 0)

That could also have been a simple hand check too. I've seen it a few times. Not agressive spidering with it though. Spot checks here and there - can't say as I blame them.

We all have different experiences. I talked with a guy today that was getting 3k referrals a day out of altavista to a modest sized site (shock me...). Same is true with Fast - I get great traffic from fast via Lycos and Sympatico.

IanTurner

WebmasterWorld Administrator ianturner us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 337 posted 7:54 am on Oct 23, 2001 (gmt 0)

Fast/Lycos is my third biggest referrer, behind MSN and Google.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved