Welcome to WebmasterWorld Guest from 54.161.95.0

Forum Moderators: Robert Charlton & aakk9999 & andy langton & goodroi

Message Too Old, No Replies

Google's Human Evaluators - 10,000 of them?

     
3:06 am on Jul 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


We've known for a while that Google uses human evaluation in evolving their algorithm. They've even got a patent for integrating the humans and the algorithm [webmasterworld.com]. On 2007-06-23, Google held a Scalability Conference in Seattle. Here's an interesting tidbit from a Q&A session with Marissa Mayer:

Q: How do they tell if they have bad results?
A: ...they have 10,000 human evaluators who are always manually checking the relevance of various results.

Article by attendee Dare Obasanjo [25hoursaday.com]

Next time we're trying to figure out some odd change in the SERPs, we might do well do remember this human factor, eh?

5:01 pm on July 12, 2007 (gmt 0)

New User

10+ Year Member

joined:Dec 2, 2004
posts:30
votes: 0


Oh Hobbs You wouldnt belive what I found in my logs from Google! :)
I'll just say this:
"We really appreciate your contribution to Ad Quality Check :-) "
:D
9:17 pm on July 12, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


Hobbs

Has the page you mentioned used to rank high for any specific popular kyewords?

9:32 pm on July 12, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member hobbs is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 19, 2004
posts:3054
votes: 3


no, not at all, it's a trivial page as I said, that's why I was hoping to find the same IP visiting more pages on my site, but obviously they are evaluating one page not the whole site. Perhaps a whole site evaluation is algo based from the sum of single page evaluations by different busy bees, which is how the bot works really, no surprises there.

I also detected the same behavior in other sessions with the evaluation referrer.

9:40 pm on July 12, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


Hobbs

Those poor evaluators are paid only $10 - $15 /hr. That doesn't buy much search quality evaluation. Google gets what its paying for :-)

9:45 pm on July 12, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member hobbs is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 19, 2004
posts:3054
votes: 3


or perhaps pay for exactly what they need ;-)
9:54 pm on July 12, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 28, 2003
posts:167
votes: 0


Btw, did Google do anything to secure the objectivity (fairness) of the evaluators work? any special requirements?

There was a test, and that was it. I was recruited with a whole bunch of Google Answers Researchers, so I guess we had already proven ourselves through our work.

9:54 pm on July 12, 2007 (gmt 0)

Senior Member

joined:Oct 27, 2001
posts:10210
votes: 0


Those poor evaluators are paid only $10 - $15 /hr. That doesn't buy much search quality evaluation. Google gets what its paying for :-)

How difficult can it be to tell #*$! from Shinola?

10:09 pm on July 12, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


EFV

How difficult can it be to tell #*$! from Shinola?

It could be rather difficult indeed. So much difficult that you need a whole bunch of 10.000 of those $10 - $15 evaluators to do the job ;-)

10:20 pm on July 12, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


robertskelton

Thanks again for feedback.

I read once that most of the evaluators/raters are young students, but that could be incorrect of course.

10:30 pm on July 12, 2007 (gmt 0)

Senior Member

joined:Oct 27, 2001
posts:10210
votes: 0


I read once that most of the evaluators/raters are young students, but that could be incorrect of course.

$10-15 per hour for rating Web sites isn't a bad job if you're a student. It beats serving up grub in the university cafeteria or unpacking cartons in the bookstore.

Also, it isn't hard to tell #*$! from Shinola if you're intelligent and have been given guidelines. My 20-year-old son (who's a university student) would be an excellent candidate for the job.

5:09 am on July 13, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Oct 19, 2004
posts:351
votes: 0


Hi,

Well 180 pages visited yesterday by a plex ip without the gbot user agent every single one of them from being supplemental to primary index and ranking where they were 3 months ago.

Hope it lasts

Vimes.

9:31 am on July 13, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 7, 2005
posts:137
votes: 0


There is a lot of talent available in Eastern Europe for 8 Euro per hour that does a lot of subcontract work for US firms.

It is not student talent.

They are very happy to get that money.

And there is a lot more that gets a lot less to do local work.

Google knows this and is opening a large operation to Eastern Europe to grab this talent.

In Eastern Europe you can get a lot more than you pay for. And Google is.

12:41 am on July 17, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 12, 2002
posts:59
votes: 0


Could this be one of them? Whilst reviewing logs I have often wondered who these guys might be. I have hundreds/ thousands visits from mountain view using various country specific google sites for the searches.

>>>>>>>>
from 72.14.192.9 in Mountain View, United States

Referred by www.google.co.uk searching for "ecommerce shopping cart"
14:48:37 Home page

6:14 pm on July 20, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member

joined:July 26, 2006
posts:1619
votes: 0


$10-15 per hour for rating Web sites

If they're outsourcing to india like everyone else then they aren't paid anywhere near that much.. not even close. Scary to think that someone in india is evaluating my pages. heck they can't even trouble shoot my dsl connection or get my dell order correct half the time. Just gives me shivers.

12:24 am on July 22, 2007 (gmt 0)

Junior Member

5+ Year Member

joined:June 19, 2007
posts:53
votes: 0


clarification:

my friend who took the job lives in LA. $12 per hour, part time, with no benefits. Where I live, ave. non-skilled work averages $7 - $8 hourly, so this is a good deal.

The job was as someone mentioned earlier: is this page spam or not? and are trained to look for certain things.

If they are hiring overseas, then probably a diff. rate.

as we all know humans can add a different perspective to a that of a computers. this is smart way for google to test effectiveness of their algorithm in the real-world.

11:35 am on Aug 8, 2007 (gmt 0)

New User

5+ Year Member

joined:Aug 8, 2007
posts:1
votes: 0


I'm planning to apply for this evaluator job because I really needed the money, and nothing have come up so far on my job search due to lack of experience. If I don't a job soon, I'll be sleeping on the street. Anyway...

1. Is it easy to get this job?
2. What type of information should I put on my cover letter and resume to increase my chances of an interview. Samples would really help.
3. Do you need to provide references for this job?
4. Do you need to take some sort of test?
5. What type of people or personality are they looking for for this job?
6. I was a webmaster for my hobbies. Should I leave this info on or off my resume?
7. How many hours of work do they give you per week?
8. Does Google check your credit background for this job?

Ok, that's a lot of questions, but any informations you can provide would really help. Thanks.

This 106 message thread spans 4 pages: 106