I don't think they shrunk the index, I think they optimized how they store and retrieve data which impacts longtail terms more.
- This keyword has been mentioned 17 times in this 2000 word article = excellent
- This keyword has been mentioned in title, uri and meta description = excellent
- This keyword has been mentioned 1 time in total on other pages of this 5000 page site = site is not about keyword, next.
Do I have proof? no. Somewhere around the midway point of the newly released hour long 2010 IO website review youtube video Matt Cutts mentions using the keyword one more time if you're on "the hairy edge" to go back to #1 from #3. I suspect this is more of a sitewide thing, GWT counts total instances of each keyword on a sitewide level and Matt confirms there is a threshold on number of times a keyword is used.
In other words, webmasters who SEO their sites to the degree that they cover each keyword combo with a separate article are out of luck on long tail keyword phrases they mention from one article only.
Sites like WW who cover them all rather frequently have a sitewide issue of being huge which works to drop the % of longtail keyword use despite being mentioned many times. The subject is covered, yes, but the site is not about that subject (in Google's eyes) and preference is given to a site that is.
Spam sites stuff related keywords all over the place which may be why they increased in rank a little (temporarily, I'm seeing them fall off again now).
What can you do? Well, if I'm right, a huge market has just opened up for sites about more longtail keyword subjects. The fact spam sites are taking up those positions now and top quality sites don't outrank them suggests I'm right. Find the spam and create solid content on the subject.
[edited by: Sgt_Kickaxe at 7:53 pm (utc) on Jun 11, 2010]