Google's Dead Zone

Alright - What Happens Between the Crawl and the Update

[Grumpus]

I'm not talking about freshbot and fresh results here, I'm just wondering about this several week period between when the deep crawl finishes and when the update begins.

Why so long? What happens over there at the Googleplex during the time when their deep crawl grows more out-of-date by the day and before they hit the update switch? Does it NEED to be that long?

Any ideas? Am I the only one who wonders what happens during this time period, which, for me, seems to be about 10-18 days longer than it needs to be?

Hmmmm...

G.

[Chris_R]

It does take a while to crawl 3 billion pages and index them.

Plus they have to compute all the links and do PR calculations.

I don't know how long each part takes, but I don't think they are letting anytime go to waste.

They have said they want to speed this up - and perhaps the fresh thing is a stop gap inbetween working out the other stuff.

[dvduval]

Grumpus,

My theory:
1) Google crawls 3,000,000,000+ pages - takes about 5 days
2) Google compares the links on each page with the links on every other page.

You can express the number of calculations involved for comparing links as a factorial: 3,083,324,652!
which is...1 times 2 times 3 times 4....times 3,083,324,652.
In other words, it's an enormous number. I'm sure Google has found some shortcuts, but when you think about it, 25 days is a short period indeed!

[Grumpus]

Yeah, I know. It does make me wonder, though. How many machines are doing the job now? Would doubling the resources thrown at the job and, thus, halving the time it takes to do it, increase their "status" enough to make it worthwhile? Obviously, they don't seem to think so - rather, they've opted to double their resources and put those to a completely different task altogether.

The crawl's beeing going on for more than 5 days, now, too. I'm still getting hit as we speak....

G.

[Iguana]

During this time they run a book on who will post the first "Update started" message on WebmasterWorld

[Yidaki]

I just spoke with my friend at Google (the one who sits left hand of the yellow button) and he said that every month between the crawl and the update the staff takes a break at hawaii. The crawler machines are shut down and the few simple daily searches are processed by the secretary's imac.

Hmm ...

;)

[Beachboy]

Well certainly what happens on OUR end of this "dead zone" is a lot of praying. And on the Big Day, we get salvation. Or whatever. ;)

[vibgyor79]

Maybe Google should use the computing power of all the Google Toolbar users (when their computer is idle) to speed up the process :)

[Chico_Loco]

Maybe Google should use the computing power of all the Google Toolbar users (when their computer is idle) to speed up the process

Now there's an idea!
Only thing is, would their trust us by sending us some of the calculation data, which someone might figure out a way to determine the PR formula from!

[europeforvisitors]

Yidaki:

I just spoke with my friend at Google (the one who sits left hand of the yellow button) and he said that every month between the crawl and the update the staff takes a break at hawaii. The crawler machines are shut down and the few simple daily searches are processed by the secretary's imac.

No, I think they've got the update process down to the point where it takes only minutes to complete. But those weekly Norton AntiVirus LiveUpdates add up to nearly a month's worth of installation time on 10,000 PCs.