BillyS - 7:49 pm on Oct 4, 2004 (gmt 0)

Here is my guess on the lag... because this is how I would want to design a system this large...

Google is maintaining multiple databases and depending on the query, it redirects the to the appropriate database. The primary database responds to a list of "common" queries and is updated less frequently than the other databases. Google does this for performance reasons - fast response, low system demand.

The secondary database responds to any query not found on the "common" query list. This database is updated more frequently than the primary database and contains a much larger set of data.

Google does this because it wants fast response to any query the end user submits. Why waste horsepower on the common stuff? It's the Pareto Principle and it makes for efficient database design.

The lag exists because Google has a threshold that a site must meet before it is contained in the primary database.

- Some sites are always in there because they beat the threshold by a long shot.
- Some sites bounce in and out because they are on the edge of acceptance.
- New sites have a hard time making the threshold.

Google wants to limit the number of pages in the primary database, so the threshold can move each month. The better the primary database, the harder it gets for new sites to gain entry.

Google even tells us how large the primary database is, it's at the bottom of their query page. That language is much better than...

Searching 4,285,199,774 web pages unless you submit an uncommon query. In that case we look to our secondary database, which holds even more web pages.

Here's more for you 32 versus 64 bit folks:
Google does this because they have a large investment in 32 bit machines and they want to use those computers. The secondary database is a 64 bit design using recently purchased machines that are more expensive and computationally more powerful. However, they do not have enough of these machines to support the sheer number of "common" queries they receive.