Google's Human Evaluators - 10,000 of them?

[tedster]

We've known for a while that Google uses human evaluation in evolving their algorithm. They've even got a patent for integrating the humans and the algorithm [webmasterworld.com]. On 2007-06-23, Google held a Scalability Conference in Seattle. Here's an interesting tidbit from a Q&A session with Marissa Mayer:

Q: How do they tell if they have bad results?
A: ...they have 10,000 human evaluators who are always manually checking the relevance of various results.

Article by attendee Dare Obasanjo [25hoursaday.com]

Next time we're trying to figure out some odd change in the SERPs, we might do well do remember this human factor, eh?

[pexcornel]

Oh Hobbs You wouldnt belive what I found in my logs from Google! :)
I'll just say this:
"We really appreciate your contribution to Ad Quality Check :-) "
:D

[reseller]

Hobbs

Has the page you mentioned used to rank high for any specific popular kyewords?

[Hobbs]

no, not at all, it's a trivial page as I said, that's why I was hoping to find the same IP visiting more pages on my site, but obviously they are evaluating one page not the whole site. Perhaps a whole site evaluation is algo based from the sum of single page evaluations by different busy bees, which is how the bot works really, no surprises there.

I also detected the same behavior in other sessions with the evaluation referrer.

[reseller]

Hobbs

Those poor evaluators are paid only $10 - $15 /hr. That doesn't buy much search quality evaluation. Google gets what its paying for :-)

[Hobbs]

or perhaps pay for exactly what they need ;-)

[robertskelton]

Btw, did Google do anything to secure the objectivity (fairness) of the evaluators work? any special requirements?

There was a test, and that was it. I was recruited with a whole bunch of Google Answers Researchers, so I guess we had already proven ourselves through our work.

[europeforvisitors]

Those poor evaluators are paid only $10 - $15 /hr. That doesn't buy much search quality evaluation. Google gets what its paying for :-)

How difficult can it be to tell #*$! from Shinola?

[reseller]

EFV

How difficult can it be to tell #*$! from Shinola?

It could be rather difficult indeed. So much difficult that you need a whole bunch of 10.000 of those $10 - $15 evaluators to do the job ;-)

[reseller]

robertskelton

Thanks again for feedback.

I read once that most of the evaluators/raters are young students, but that could be incorrect of course.

[europeforvisitors]

I read once that most of the evaluators/raters are young students, but that could be incorrect of course.

$10-15 per hour for rating Web sites isn't a bad job if you're a student. It beats serving up grub in the university cafeteria or unpacking cartons in the bookstore.

Also, it isn't hard to tell #*$! from Shinola if you're intelligent and have been given guidelines. My 20-year-old son (who's a university student) would be an excellent candidate for the job.

[Vimes]

Hi,

Well 180 pages visited yesterday by a plex ip without the gbot user agent every single one of them from being supplemental to primary index and ranking where they were 3 months ago.

Hope it lasts

Vimes.

[Gimp]

There is a lot of talent available in Eastern Europe for 8 Euro per hour that does a lot of subcontract work for US firms.

It is not student talent.

They are very happy to get that money.

And there is a lot more that gets a lot less to do local work.

Google knows this and is opening a large operation to Eastern Europe to grab this talent.

In Eastern Europe you can get a lot more than you pay for. And Google is.

[Fighting Falcon]

Could this be one of them? Whilst reviewing logs I have often wondered who these guys might be. I have hundreds/ thousands visits from mountain view using various country specific google sites for the searches.

>>>>>>>>
from 72.14.192.9 in Mountain View, United States

Referred by www.google.co.uk searching for "ecommerce shopping cart"
14:48:37 Home page

[Bewenched]

$10-15 per hour for rating Web sites

If they're outsourcing to india like everyone else then they aren't paid anywhere near that much.. not even close. Scary to think that someone in india is evaluating my pages. heck they can't even trouble shoot my dsl connection or get my dell order correct half the time. Just gives me shivers.

[hippypink]

clarification:

my friend who took the job lives in LA. $12 per hour, part time, with no benefits. Where I live, ave. non-skilled work averages $7 - $8 hourly, so this is a good deal.

The job was as someone mentioned earlier: is this page spam or not? and are trained to look for certain things.

If they are hiring overseas, then probably a diff. rate.

as we all know humans can add a different perspective to a that of a computers. this is smart way for google to test effectiveness of their algorithm in the real-world.

[plasticstyles]

I'm planning to apply for this evaluator job because I really needed the money, and nothing have come up so far on my job search due to lack of experience. If I don't a job soon, I'll be sleeping on the street. Anyway...

1. Is it easy to get this job?
2. What type of information should I put on my cover letter and resume to increase my chances of an interview. Samples would really help.
3. Do you need to provide references for this job?
4. Do you need to take some sort of test?
5. What type of people or personality are they looking for for this job?
6. I was a webmaster for my hobbies. Should I leave this info on or off my resume?
7. How many hours of work do they give you per week?
8. Does Google check your credit background for this job?

Ok, that's a lot of questions, but any informations you can provide would really help. Thanks.