Welcome to WebmasterWorld Guest from 54.225.31.78

Forum Moderators: open

Message Too Old, No Replies

fast spidering with libwww user agent

     
5:17 am on Oct 23, 2001 (gmt 0)

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member



fpcr001.sac2.fastsearch.net - - [22/Oct/2001:21:39:31 -0400] "GET /mypage.html HTTP/1.0" 200 8360 "-" "libwww-perl/5.
52 FP/4.0"
fpcr001.sac2.fastsearch.net - - [22/Oct/2001:21:39:38 -0400] "GET /the-other-one.html HTTP/1.0" 200 8241 "-" "libwww-perl/5.52 FP/4.0"

What I'm wondering is does this mean that they forgot (when they were going around incognito) to check for the "if modified since" date on the pages in question, or when they are doing the 'cloak busting' thing, they just ignore that...

Just curious. It's funny to see spiders hit a site more times than they send referrals...even with good rankings.

7:18 am on Oct 23, 2001 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



That could also have been a simple hand check too. I've seen it a few times. Not agressive spidering with it though. Spot checks here and there - can't say as I blame them.

We all have different experiences. I talked with a guy today that was getting 3k referrals a day out of altavista to a modest sized site (shock me...). Same is true with Fast - I get great traffic from fast via Lycos and Sympatico.

7:54 am on Oct 23, 2001 (gmt 0)

WebmasterWorld Administrator ianturner is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Fast/Lycos is my third biggest referrer, behind MSN and Google.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month