fast spider reading http code 304

Forum Moderators: open

Message Too Old, No Replies

fast spider reading http code 304

jeremy goodrich

5:34 pm on Sep 28, 2001 (gmt 0)

That is the apache not modified since last visit header, correct? I haven't actually bothered with their spider much recently...though they do send a tiny bit of traffic.

How long have they been looking at things other than 200? This is news to me...and I looked through the forum for a few minutes before posting, didn't notice anyone else mentioning this...makes sense that they are doing it, to keep a fresh database of info, and not bothering to reprocess the same exact data twice.

Anyone else seeing this?

littleman

6:08 pm on Sep 28, 2001 (gmt 0)

The way it works is Fast's bot comes in with a IF_MODIFIED_SINCE = some date and then it is up to the server to return the page or serve up the error message. The bot doesn't know what is going to be returned, the server checks for the IF_MODIFIED_SINCE header and makes the call.

stephen baker

9:24 pm on Sep 28, 2001 (gmt 0)

Hey there. This is Stephen from FAST. Stupid question for you...when you say that we send you a tiny bit of traffic, are you referring to AllTheWeb only, or to Lycos, SOL, Wind, etc..all of the other portals that we power? Thanks!

jeremy goodrich

4:44 am on Oct 1, 2001 (gmt 0)

:) Must apologize for that, Stephen from FAST..

What I meant was that my personal search engine related site has only recieved, since inception, 2 clicks from Fast Search and Transfer powered search engines and portals. Then again, it's only been around since June.

Sorry, but it seems that people don't search much for what I wrote within your database. Or I'm doing something wrong, and optimizing poorly :) One of the two referrals I recieved was actually from Lycos. It was quite flattering, because I didn't think my site would come up in a search so, how do I say, scientific?.

Then again, the site has only recieved referrals from Google, Alltheweb, Lycos, AOL, DMOZ, Wisenut and partners who have linked to it. So actually, being last out of 6 isn't too bad, considering Inktomi, Altavista, Excite, etc. have never sent a single referral, nor have Northernlight, IWon, directhit, ask jeeves, teoma, looksmart, goto, findwhat...etc.

BTW, IMHO, the stupid questions are the ones we don't ask. Cheers.