Welcome to WebmasterWorld Guest from 54.234.153.197

Forum Moderators: Robert Charlton & aakk9999 & andy langton & goodroi

Message Too Old, No Replies

Google's Human Evaluators - 10,000 of them?

     
3:06 am on Jul 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


We've known for a while that Google uses human evaluation in evolving their algorithm. They've even got a patent for integrating the humans and the algorithm [webmasterworld.com]. On 2007-06-23, Google held a Scalability Conference in Seattle. Here's an interesting tidbit from a Q&A session with Marissa Mayer:

Q: How do they tell if they have bad results?
A: ...they have 10,000 human evaluators who are always manually checking the relevance of various results.

Article by attendee Dare Obasanjo [25hoursaday.com]

Next time we're trying to figure out some odd change in the SERPs, we might do well do remember this human factor, eh?

5:32 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member hobbs is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 19, 2004
posts:3054
votes: 3


What worries me is the question if those evaluators are civilians (non owners of web sites)

I'd hate to compete in the serps with one of those with all the inside information and connections they have, sort of like dmoz all over again.

5:44 pm on July 10, 2007 (gmt 0)

Senior Member

joined:Oct 27, 2001
posts:10210
votes: 0


I don't get the impression, from public statements made by GoogleGuy and others, that Google's quality evaluators are being used to "clean out" search results. There's no reason to believe that they have the power of DMOZ editors, who act as gatekeepers for their assigned DMOZ topics.
5:45 pm on July 10, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 19, 2006
posts:508
votes: 0


Even if they do have 10,000 quality evaluators, a lot of those people may be (and probably should be) part-timers.

I think most of them don't even know they are considered human evaluators by Google... ;)

[edited by: wildbest at 5:46 pm (utc) on July 10, 2007]

5:45 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 3, 2002
posts:2575
votes: 0


10,000 human evaluators

Do these 10,000 evaluators have the power to make manual serp changes? If so, that is scary.

sort of like dmoz all over again

My thoughts exactly.

5:48 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


Do these 10,000 evaluators have the power to make manual serp changes?

Not according to the patent [webmasterworld.com]. You're right, that would be a scary scenario.

5:53 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


nickreynolds

Any guesses as to the salary?

According to this article [searchbistro.com]:

The Google testers are paid $10 - $20 for each hour they filter the results of Google. Payment is done through Payroll. The international agents are instructed by the Human Quality Evaluation Team of Google.
5:58 pm on July 10, 2007 (gmt 0)

Full Member

10+ Year Member

joined:May 13, 2005
posts:240
votes: 0


Nobody in this thread has mentioned a visit to their website by a human evaluator.

I did have visits from them on a new site something like 2-3-4 months ago. I spotted them as I was monitoring the logs of the new site. I recall seeing something like 5 visits from various computers based in my country, and the referrer contained something like G evaluation.

My country (former Soviet bloc) is not a big one, population is about 1/30 of that of the US. If my local language site could receive 5 evaluators (probably not everyone in G's local team visited my site), the number of staff in larger markets might be magnitudes larger.

6:17 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 20, 2004
posts:2377
votes: 0


If I remember correctly, I was paid around $15/hr.

To answer the question about being able to change SERPS, that was not an option.. =)

We were told our data would be used for research purposes and would not directly affect the search results.

It is my opinion that our rating data was/is used to help better tweak their algorithm.

6:29 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


maximillianos

Did you get the job at that time through a similar job announcement like this one?

QUALITY RATER - (SPANISH, DUTCH, ITALIAN, FRENCH) This is a temporary role offered through Kelly Services. Google Inc. is recruiting part-time, temporary, home-based workers to help with work on a search quality evaluation on a project basis. You would work at your own pace, and the time and length of any particular work session would be up to you. Candidates will evaluate search results and rate their relevance....

< I shortened the quote
see Terms of Service [webmasterworld.com] >

[edited by: tedster at 7:49 pm (utc) on July 10, 2007]

6:30 pm on July 10, 2007 (gmt 0)

Junior Member

joined:July 4, 2005
posts:85
votes: 0


Google employs over 10,000 full time people and has temporary employees as well. I would consider every one of them a "human evaluator" that has an impact on search results.
6:33 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 20, 2004
posts:2377
votes: 0


Did you get the job at that time through a similar job announcement like this one?

reseller - Yes, that look pretty much exactly like the job ad I responded to.

6:40 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 1, 2004
posts:3181
votes: 0


>>Google employs over 10,000 full time people and has temporary employees as well. I would consider every one of them a "human evaluator" that has an impact on search results.

They don't want to listen to us kkobashi. I think you, Brett, and I are all on the same page.

6:49 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


maximillianos

Thanks for feedback. Much appreciated.

But it seems they weren't interested at all whether the Raters themselves are webmasters owning for example business sites or are part of affiliate programs. I mean Google didn't care about the objectivity (fairness) of the Raters.

6:57 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 20, 2004
posts:2377
votes: 0


But it seems they weren't interested at all whether the Raters themselves are webmasters owning for example business sites or are part of affiliate programs. I mean Google didn't care about the objectivity (fairness) of the Raters.

I did have to jump through hoops to get the job since I was a webmaster. I had to get some sort of special approval where they reviewed my accounts with G and my websites, etc. From that info, I do believe they filtered the type of sites I could/would be rating. Not positive about that...

6:58 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 20, 2004
posts:2377
votes: 0


BTW - the interview/testing process is a pain. It takes about 10 hours of your time... but you do get paid for it.

They are very thorough.

6:59 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


BillyS

They don't want to listen to us kkobashi. I think you, Brett, and I are all on the same page.

The problems with your argument are:

- We are not sure whether Marissa Mayer was referring to Google's 10.000 employees.

- Taking into account the poor quality of Google's serps in specific sectors until recently. And taking into account that 10.000 highly educated Google employees were doing the evaluation job. I wouldn't consider that as a good PR for the folks at the plex.

Better to balme it on the $10 raters ;-)

7:07 pm on July 10, 2007 (gmt 0)

Junior Member

joined:July 4, 2005
posts:85
votes: 0


BillyS: Yeah and I'm sure all of those 10,000 outsourced India temps know how to read Chinese websites too :o)
7:39 pm on July 10, 2007 (gmt 0)

Junior Member

5+ Year Member

joined:Sept 10, 2006
posts:47
votes: 0


I see the side of this where people may feel misjudged on why a site ranks low or high, banned or not. But I also see the side where there can be some justification of having borderline techniques for Ajax and Flash sites doing image replacement, flash replacement, essentially are simply hidden text but not made to stuff with keywords. Having some human check it out can see if the purpose is really a bad purpose or not.
7:44 pm on July 10, 2007 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member ogletree is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 14, 2003
posts:4267
votes: 21


They also hire humans to read blogs and forums to see what webmasters are doing and talking about. They work in Seattle.
7:54 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 20, 2004
posts:2377
votes: 0


They also hire humans to read blogs and forums to see what webmasters are doing and talking about. They work in Seattle.

They are watching us right now! ;-)

8:12 pm on July 10, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Oct 21, 2003
posts:354
votes: 0


Being living in India and having several friends in Hydrabad working for big IT companies, I havent ever heard somebody working as an serach evaluator or whatever with Google... part time or full time.

Do they impose so harsh non disclosure policy?

There have to be many ex-evaluators who write blogs or post on Forums ;-)

8:34 pm on July 10, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 2, 2005
posts:52
votes: 0


"In order to storm Rohan we would need thousands."

"tens of thousands".

"but my lord, there is no such army..."

Cut to the news.

8:40 pm on July 10, 2007 (gmt 0)

Moderator

WebmasterWorld Administrator skibum is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Sept 20, 2000
posts:4470
votes: 1


According to the ultimate Google authority (Wiki), they had 12,238 full time employees as of March 31, 2007 so virtually all of them would have to be pt/temp and if they are how could you count on them for QC?

Wasn't not doing hand manipulations some reason they put out in court when someone to dogde a lawsuit when somebody lost their rankings before? Maybe I've got that wrong...

8:52 pm on July 10, 2007 (gmt 0)

Senior Member

joined:Oct 27, 2001
posts:10210
votes: 0


BTW - the interview/testing process is a pain. It takes about 10 hours of your time... but you do get paid for it.

They are very thorough.

Sounds like they aren't stupid, either.

Wasn't not doing hand manipulations some reason they put out in court when someone to dogde a lawsuit when somebody lost their rankings before? Maybe I've got that wrong...

You're probably thinking of the SearchKing case (which SearchKing lost), but I don't believe Google has ever claimed that it doesn't do manual penalties. Indeed, the court decision reaffirmed Google's first-amendment right to adjust rankings or remove pages from its search results. Still, that isn't directly related to what we're talking about here, since--as we've just been told--Google's quality evaluators aren't doling out penalties or manipulating rankings.

9:07 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


Folks!

This whole human-role-in-google-search thing is a farse, IMO.

Google has been preaching algo driven serps for years. Then few semi-human-edited and fully-human-edited-search-engines emerged.

Mighty GOOG got worried, and thought; lets jump on the human-edited-search-engines wagon.

And suddenly we see the kind Googlers busy telling the world; Ooooops...we forgot to tell ya that Google in fact has always been a human-edited-search-engine.

The remaining question to be answered by GOOG is: does the quality of human-edited-serps better than that of algo-driven-serps?

Good night and God bless.

10:24 pm on July 10, 2007 (gmt 0)

Junior Member

5+ Year Member

joined:June 20, 2007
posts:54
votes: 0


10,000 silent OompaLoompa's perhaps?

Nobody ever goes in, nobody ever comes out... :-)

10:29 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 12, 2003
posts:850
votes: 0


I know a woman who works as an independent contractor for Google, in search quality. About the only thing she'll say about the work is that she signed a non-disclosure agreement.
10:44 pm on July 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 29, 2003
posts:790
votes: 0


correct me if I'm wrong,

but 3 hors daily @ 15$ * 200 days * 10000 people = 90 million dollars per year - that's small change for a giant like Google, even if you add 50% on top of that for recruiting services and the like.

10:47 pm on July 10, 2007 (gmt 0)

Senior Member

joined:Oct 27, 2001
posts:10210
votes: 0


This whole human-role-in-google-search thing is a farse, IMO. Google has been preaching algo driven serps for years.

Search engines don't design themselves. Humans provide basic parameters and supply input, whether directly or via examples of relevant and irrelevant or good and bad pages for a "black box." Do a search for GoogleGuy's comments on how human evaluators are used, and re-read what maximillianos a few hours ago:

"We were told our data would be used for research purposes and would not directly affect the search results."

It's also worth noting that PageRank, which was one of Google's founding principles, requires human input. (Humans, not algorithms, create the links that serve as the "votes" on which PageRank is based.)

11:13 pm on July 10, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:July 12, 2004
posts:384
votes: 0


Is it me or is this suggestion absolutely ridiculous in the extreme!

To base your whole business model on algorithmic technology which was the whole reason people used you in the first place - lets be honest anything apart from search has been a huge, huge, huge (revenue) failure for Google. And then employ 10k individuals to "validate" and provide data to filter the algorithm.

What the hell are they doing then, the results have not improved - they have not removed spam, more legitimised it through mass ebay, amazon, wikipedia, shopping comparison, holding domains, mfa - etc. etc. We all know that if you are ebay then the rules are different than a small site that has missing in action for months or even years - I know that because I have been on the other side of it where I got to speak (on behalf of my rather large company in my capacity of technical director) to Google where they advised on how to correct indexing and ranking problems and within hours it was sorted (we are talking millions of pages).

What do these 10k people do - try various keyphrases to check the results? Try bloody acting on what webmasters in this forum have been talking about for years - 301s, URL hijacking, shallow crawling, incorrect duplicate penalties etc etc. And that is for free!

It is lucky I now treat Google with the contempt it deserves and now think absolutely nothing about it other than to make money out of their search results (that is my job as a webmaster) - I once gave a damn about ethics and "growth", "building", "quality", "longevity" and "trust", now it is so crazily volatile that I have to give a damn about me. I can't now call up Google to ask them to sort my indexing and ranking anymore as I am on my own now - just like everyone else who is not part of a huge site delivering millions of dollars to Google.

This 106 message thread spans 4 pages: 106
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members