tedster - 9:00 pm on Oct 20, 2011 (gmt 0)
Do they grab say the top 20 websites in the SERP for all say...search terms over 50k searches a month and let the EWOQ team classify these sites?
I'm sure that your description is one kind of project that is created and assigned to human quality raters. It's very much the kind of thing that the 2006 patent described. Quality raters are assessing the SERP itself.
In that regard, I think they may often be evaluating a proposed algorithm tweak in a kind of experimental mode - before it ever gets even a low level live test. Somewhere recently I read that something like 5,000 changes get proposed every year, only 500 of which actually get pushed into production.