dusky - 11:22 pm on May 1, 2010 (gmt 0)
IMO, this is what's probably happening (I said this before), half of the data is on the new IS, the other half is still on the old or vice-versa, the algo is pointing and trying to retrieve / fetch search queries results from the new IS, however, half of the data is still somewhere else, so it fetches results based on what the databases have, and they have a lot less. The data that is still to be moved and awarded PR and IBLs/EBLs is mostly deep end-of-branch pages which are the bulk of the Internet, while homepages and second in PR command pages are likely to be the short tail well established pages, and they are mostly fine because they were moved first.
I wrote a crawler once while back and when doing the crawling, I fed it first the authority sites homepages mostly from Y and Dmoz directories in addition to Alexa's top 100k, and let it work and drill downwards through them before fetching links posted on them. Feeding that into my infrastructure was by page hierarchy importance, the deeper the page, the longer it takes for the crawler-indexer to work out where to throw / place it, hence if one were to search that particular site, they will get results that are not necessarily worthy of user's satisfaction, because better results aren't yet inserted into the database and would take days, so my small-man algo at the time did not have any choice, you offer answers based on what you find as records. OK, those were the days when you can have 100m index and think you are a big shot, if the scenario is true for sure on G*s case, no surprise it's taking months and getting more and more noticeable, happening to more sites, the whacking is even harder now because we are in the middle of the MAYDAY update which can still go on for longer. MAYDAY is certainly the milestone as more people than ever felt the heat, just like BD and Florida, though those were not one day dump-all either!
This is why I think is deliberate and they can't wiggle their way out of it, it's the sacrifice to pay (we pay, to probably make them pay) for moving to larger IS.