Phil_Payne - 8:29 pm on May 3, 2006 (gmt 0)

> When faced with these sort of storage problems you need to keep a check on the refresh rate of your data - especially if your algo has a "time" factor i.e. link rate, domain history etc. because then your data storage requirement explodes.

True. I was for quite a time involved in large systems performance (Vice-Chairman of UK CMG) and nasty things happen in queueing theory when resources get tight - the "knee of the curve" effect. If Google is tight on storage, it will be spending ever more time and resources finding places to stick things. Rules of thumb (crude, I know) suggest 70% is about the best utilisation of a storage resource you'll ever get if you want the system to perform.

I've commented in other places about a site of mine where Google's response to my sitemap updates has been odd. I've now realised exactly what's happening - my Googlebot is faithfully downloading changes I notified via new sitemaps around five weeks ago. It's just worked its way through March 28 and started on March 29 - in order.

The corollary is that Google is delivering results to users based on old data. At least five weeks old, on this one site. It won't take long for Joe Public to start to realise that Yahoo, Ask Jeeves, etc., deliver more pertinent results than Google. In some consumer markets, five weeks is the lifetime of a product.

[edited by: Phil_Payne at 8:47 pm (utc) on May 3, 2006]