The Freshbot Curse

I'm dyin' here.

[twilight47]

Oh, this wretched machine that raises me to such highs and then pulls the magic carpet out from under my feet of clay... or is it that I fly to close to the sun and my melted wings cause me to plummet to the earth ... well whichever metaphor you choose.

Freshbot has been indexing our site all this last month, four time our freshened and cached homepage appears at the tops of SERPs. Hits and orders triple. All is right with the universe. And then a few days later, <PooF>, like Keyser Soze, vanished from existence. The only things left over are old cached pages like a body in the desert pick over by vultures.

Needless to say that with no updated optimized index page, sales fall off, bridges are lept from, and widows are left to collect the insurance money.
All of this, only to happen all again a few days later like some causal time loop instigated by a massive energy fluctuation in a rift in the space/time continuum.

When will the madness end?
Will this stop after the update?
Will post update rankings be similar to the freshbot ones?

...and what about Naomi?

[BigDave]

You should make sure that you implement if-modified-since and never update your pages.

Then freshbot will never visit, and you won't get those annoying spikes in sales that lead to later depression. You will have those nice steady low sales that you want.

[deanril]

Bigdave can you explain further? Modified since?

You men those things on sits that say this page was last modified, and there is a date? Does this keep freshbot coming?

[BigDave]

When you get a page from a site, it also returns the last modified date for that page. The next time googlebot requests the page is sends "get if-modified-since" the date that was returned the last time it got the page. If it has not been modified, your server will return the status code "not modified" and google will use the old version. It is a way of saving bandwidth.

[Krapulator]

Check this thread Dearil: [webmasterworld.com...]

[deanril]

Thank you I understand now!

[icarus]

hello twighlight,

I see you know the origins behind my nick - it's to try and remind me not to be like the character I guess :)

Use those fresh listings as a guide for future improvements, but don't push it too far - PR0 is *really* losing your wings.. low rankings are just a few feathers gone that can grow back.