Welcome to WebmasterWorld Guest from 54.167.46.29

Forum Moderators: open

Message Too Old, No Replies

fast spidering with libwww user agent

     
5:17 am on Oct 23, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 4, 2000
posts:3468
votes: 0


fpcr001.sac2.fastsearch.net - - [22/Oct/2001:21:39:31 -0400] "GET /mypage.html HTTP/1.0" 200 8360 "-" "libwww-perl/5.
52 FP/4.0"
fpcr001.sac2.fastsearch.net - - [22/Oct/2001:21:39:38 -0400] "GET /the-other-one.html HTTP/1.0" 200 8241 "-" "libwww-perl/5.52 FP/4.0"

What I'm wondering is does this mean that they forgot (when they were going around incognito) to check for the "if modified since" date on the pages in question, or when they are doing the 'cloak busting' thing, they just ignore that...

Just curious. It's funny to see spiders hit a site more times than they send referrals...even with good rankings.

7:18 am on Oct 23, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


That could also have been a simple hand check too. I've seen it a few times. Not agressive spidering with it though. Spot checks here and there - can't say as I blame them.

We all have different experiences. I talked with a guy today that was getting 3k referrals a day out of altavista to a modest sized site (shock me...). Same is true with Fast - I get great traffic from fast via Lycos and Sympatico.

7:54 am on Oct 23, 2001 (gmt 0)

Moderator from GB 

WebmasterWorld Administrator ianturner is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 19, 2001
posts:3452
votes: 10


Fast/Lycos is my third biggest referrer, behind MSN and Google.