Swanson - 7:53 pm on May 3, 2006 (gmt 0)

Ha ha - priceless!

I don't think anyone could sum up Google's problem better than Webmasterworld - just look at a few of the main forum topics at the moment:

1. "Google CEO admits, "We have a huge machine crisis"
2. "Pages Dropping Out of Big Daddy Index"
3. "Somethings Up Right Now! -- 30 domains just went "home page only"

At the end of the day, in some part it can explain some of the things that are going on (including the new caching proxy where multiple crawlers cache one view of the data).

To me, if you have a huge space problem you solve it in 2 ways - more space, less data. Now then, if you come up with an amazing technique to store less data (i.e. duplicate content removal, canonical improvements) then you have saved a huge amount of cash, but.... if you underestimate what the implications are of your assumptions, you removed millions of web pages by accident. And better yet, if you do it "live" you have a bigger problem - but you have no choice but to do it live because you are running out of space.

But, you have loads of other services that need data storage - which ones do you sacrifice (the free ones or the ones that generate your whole income). Thats right, it is a better business decision to compromise your free search product (nobody may notice after all if you do it right) than your paid one (adsense and adwords need the space and no compromises can be made on this).

And there is the difference between Google of 1997 and the Google of today.