Supplemental Index - it's still alive

[SEOPTI]

Will Google ever fix their supplemental index and make it searchable? I think they should bring back the supplemental tag. Hiding it is not a solution.

Most webmasters have a fraction, maybe 1-2% of URLs in the main index.

[SEOPTI]

Just imaginge their whole index and probably 90% - 98% of their index is compressed and therefore not searchable. This is huge!

They seem to still have major data storing problems and need to compress most parts of their index.

[tedster]

I don't see the Supplemental Index exactly as a data storage fix - more like fix for data retrieval speed. By shuffling off urls to the Supplemental Index when Google's algo show them as unlikely to be tapped for a common query, the speedy return of an initial result set (1,000 or fewer urls) for a second level of filtering and re-ranking can go much quicker.

The Supplemental Index will not go away - it's an integral part of Google's infrastructure now. It seems as though this Google patent [patft.uspto.gov] which describes a "System and method for selectively searching partitions of a database", gives us such a picture - in fact, a picture where the "Supplemental Index" may soon be (or already has been?) replaced by more than one secondary partition.

No doubt that continued growth of the web puts intense demands on the need for quick and useful search results. The Supplemental Index(es?) look to me like one way Google is coping with the need for continued scalability of their infrastructure.

The historical downside we've seen for the website owner is this: yes, the full document for a "supplemental url" was cached somewhere by Google and visible as a cached page. But its complete content still did NOT seem to be fully indexed and searchable. Only the top level relevance indicators from the url, such as the title element, were available for building the search results page.

I have hopes that this limiting factor is changing, even if the spidering frequency for supplemental urls stays low. When the "Supplemental Result" tags were dropped from the SERP last summer, Google did talk about future improvements, and this is the main imporvement that I hope for.

Here's another negative factor for the site owner of the supplemental url. Results from a supplemental index or "extended index" are only tapped at all if the number of intial responses to the query from the main index falls below a certain threshold number. This makes for a much quicker retrieval for common search results, but it also makes it very unlikely that a supplemental url will be returned for a common search. I guess we already know that from practical experience, right?

If all this technology is functioning as intended (the threshold measures and so on), then those supplemental urls would not be returned for common search terms anyway. So for me, the hoped for improvement is definitely a complete indexing for the content of a supplemental url.

[SEOPTI]

So when will Google finally offer a complete indexing of supplemental URLs? I've been searching some webmaster forums but no one knows for sure. It's of course easier for them to remove the supplemental tag from their index instead fixing this problem.