homepage Welcome to WebmasterWorld Guest from 54.205.207.53
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

This 106 message thread spans 4 pages: < < 106 ( 1 [2] 3 4 > >     
Google's Human Evaluators - 10,000 of them?
tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3390143 posted 3:06 am on Jul 10, 2007 (gmt 0)

We've known for a while that Google uses human evaluation in evolving their algorithm. They've even got a patent for integrating the humans and the algorithm [webmasterworld.com]. On 2007-06-23, Google held a Scalability Conference in Seattle. Here's an interesting tidbit from a Q&A session with Marissa Mayer:

Q: How do they tell if they have bad results?
A: ...they have 10,000 human evaluators who are always manually checking the relevance of various results.

Article by attendee Dare Obasanjo [25hoursaday.com]

Next time we're trying to figure out some odd change in the SERPs, we might do well do remember this human factor, eh?

 

Hobbs

WebmasterWorld Senior Member hobbs us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3390143 posted 5:32 pm on Jul 10, 2007 (gmt 0)

What worries me is the question if those evaluators are civilians (non owners of web sites)

I'd hate to compete in the serps with one of those with all the inside information and connections they have, sort of like dmoz all over again.

europeforvisitors



 
Msg#: 3390143 posted 5:44 pm on Jul 10, 2007 (gmt 0)

I don't get the impression, from public statements made by GoogleGuy and others, that Google's quality evaluators are being used to "clean out" search results. There's no reason to believe that they have the power of DMOZ editors, who act as gatekeepers for their assigned DMOZ topics.

wildbest

5+ Year Member



 
Msg#: 3390143 posted 5:45 pm on Jul 10, 2007 (gmt 0)

Even if they do have 10,000 quality evaluators, a lot of those people may be (and probably should be) part-timers.

I think most of them don't even know they are considered human evaluators by Google... ;)

[edited by: wildbest at 5:46 pm (utc) on July 10, 2007]

crobb305

WebmasterWorld Senior Member crobb305 us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3390143 posted 5:45 pm on Jul 10, 2007 (gmt 0)

10,000 human evaluators

Do these 10,000 evaluators have the power to make manual serp changes? If so, that is scary.

sort of like dmoz all over again

My thoughts exactly.

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3390143 posted 5:48 pm on Jul 10, 2007 (gmt 0)

Do these 10,000 evaluators have the power to make manual serp changes?

Not according to the patent [webmasterworld.com]. You're right, that would be a scary scenario.

reseller

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 5:53 pm on Jul 10, 2007 (gmt 0)

nickreynolds

Any guesses as to the salary?

According to this article [searchbistro.com]:

The Google testers are paid $10 - $20 for each hour they filter the results of Google. Payment is done through Payroll. The international agents are instructed by the Human Quality Evaluation Team of Google.

radix

5+ Year Member



 
Msg#: 3390143 posted 5:58 pm on Jul 10, 2007 (gmt 0)

Nobody in this thread has mentioned a visit to their website by a human evaluator.

I did have visits from them on a new site something like 2-3-4 months ago. I spotted them as I was monitoring the logs of the new site. I recall seeing something like 5 visits from various computers based in my country, and the referrer contained something like G evaluation.

My country (former Soviet bloc) is not a big one, population is about 1/30 of that of the US. If my local language site could receive 5 evaluators (probably not everyone in G's local team visited my site), the number of staff in larger markets might be magnitudes larger.

maximillianos

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 6:17 pm on Jul 10, 2007 (gmt 0)

If I remember correctly, I was paid around $15/hr.

To answer the question about being able to change SERPS, that was not an option.. =)

We were told our data would be used for research purposes and would not directly affect the search results.

It is my opinion that our rating data was/is used to help better tweak their algorithm.

reseller

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 6:29 pm on Jul 10, 2007 (gmt 0)

maximillianos

Did you get the job at that time through a similar job announcement like this one?

QUALITY RATER - (SPANISH, DUTCH, ITALIAN, FRENCH) This is a temporary role offered through Kelly Services. Google Inc. is recruiting part-time, temporary, home-based workers to help with work on a search quality evaluation on a project basis. You would work at your own pace, and the time and length of any particular work session would be up to you. Candidates will evaluate search results and rate their relevance....

< I shortened the quote
see Terms of Service [webmasterworld.com] >

[edited by: tedster at 7:49 pm (utc) on July 10, 2007]

kkobashi



 
Msg#: 3390143 posted 6:30 pm on Jul 10, 2007 (gmt 0)

Google employs over 10,000 full time people and has temporary employees as well. I would consider every one of them a "human evaluator" that has an impact on search results.

maximillianos

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 6:33 pm on Jul 10, 2007 (gmt 0)

Did you get the job at that time through a similar job announcement like this one?

reseller - Yes, that look pretty much exactly like the job ad I responded to.

BillyS

WebmasterWorld Senior Member billys us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3390143 posted 6:40 pm on Jul 10, 2007 (gmt 0)

>>Google employs over 10,000 full time people and has temporary employees as well. I would consider every one of them a "human evaluator" that has an impact on search results.

They don't want to listen to us kkobashi. I think you, Brett, and I are all on the same page.

reseller

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 6:49 pm on Jul 10, 2007 (gmt 0)

maximillianos

Thanks for feedback. Much appreciated.

But it seems they weren't interested at all whether the Raters themselves are webmasters owning for example business sites or are part of affiliate programs. I mean Google didn't care about the objectivity (fairness) of the Raters.

maximillianos

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 6:57 pm on Jul 10, 2007 (gmt 0)

But it seems they weren't interested at all whether the Raters themselves are webmasters owning for example business sites or are part of affiliate programs. I mean Google didn't care about the objectivity (fairness) of the Raters.

I did have to jump through hoops to get the job since I was a webmaster. I had to get some sort of special approval where they reviewed my accounts with G and my websites, etc. From that info, I do believe they filtered the type of sites I could/would be rating. Not positive about that...

maximillianos

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 6:58 pm on Jul 10, 2007 (gmt 0)

BTW - the interview/testing process is a pain. It takes about 10 hours of your time... but you do get paid for it.

They are very thorough.

reseller

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 6:59 pm on Jul 10, 2007 (gmt 0)

BillyS

They don't want to listen to us kkobashi. I think you, Brett, and I are all on the same page.

The problems with your argument are:

- We are not sure whether Marissa Mayer was referring to Google's 10.000 employees.

- Taking into account the poor quality of Google's serps in specific sectors until recently. And taking into account that 10.000 highly educated Google employees were doing the evaluation job. I wouldn't consider that as a good PR for the folks at the plex.

Better to balme it on the $10 raters ;-)

kkobashi



 
Msg#: 3390143 posted 7:07 pm on Jul 10, 2007 (gmt 0)

BillyS: Yeah and I'm sure all of those 10,000 outsourced India temps know how to read Chinese websites too :o)

BenjArriola

5+ Year Member



 
Msg#: 3390143 posted 7:39 pm on Jul 10, 2007 (gmt 0)

I see the side of this where people may feel misjudged on why a site ranks low or high, banned or not. But I also see the side where there can be some justification of having borderline techniques for Ajax and Flash sites doing image replacement, flash replacement, essentially are simply hidden text but not made to stuff with keywords. Having some human check it out can see if the purpose is really a bad purpose or not.

ogletree

WebmasterWorld Senior Member ogletree us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3390143 posted 7:44 pm on Jul 10, 2007 (gmt 0)

They also hire humans to read blogs and forums to see what webmasters are doing and talking about. They work in Seattle.

maximillianos

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 7:54 pm on Jul 10, 2007 (gmt 0)

They also hire humans to read blogs and forums to see what webmasters are doing and talking about. They work in Seattle.

They are watching us right now! ;-)

wanna_learn

10+ Year Member



 
Msg#: 3390143 posted 8:12 pm on Jul 10, 2007 (gmt 0)

Being living in India and having several friends in Hydrabad working for big IT companies, I havent ever heard somebody working as an serach evaluator or whatever with Google... part time or full time.

Do they impose so harsh non disclosure policy?

There have to be many ex-evaluators who write blogs or post on Forums ;-)

rarethings55

5+ Year Member



 
Msg#: 3390143 posted 8:34 pm on Jul 10, 2007 (gmt 0)

"In order to storm Rohan we would need thousands."

"tens of thousands".

"but my lord, there is no such army..."

Cut to the news.

skibum

WebmasterWorld Administrator skibum us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3390143 posted 8:40 pm on Jul 10, 2007 (gmt 0)

According to the ultimate Google authority (Wiki), they had 12,238 full time employees as of March 31, 2007 so virtually all of them would have to be pt/temp and if they are how could you count on them for QC?

Wasn't not doing hand manipulations some reason they put out in court when someone to dogde a lawsuit when somebody lost their rankings before? Maybe I've got that wrong...

europeforvisitors



 
Msg#: 3390143 posted 8:52 pm on Jul 10, 2007 (gmt 0)

BTW - the interview/testing process is a pain. It takes about 10 hours of your time... but you do get paid for it.

They are very thorough.

Sounds like they aren't stupid, either.

Wasn't not doing hand manipulations some reason they put out in court when someone to dogde a lawsuit when somebody lost their rankings before? Maybe I've got that wrong...

You're probably thinking of the SearchKing case (which SearchKing lost), but I don't believe Google has ever claimed that it doesn't do manual penalties. Indeed, the court decision reaffirmed Google's first-amendment right to adjust rankings or remove pages from its search results. Still, that isn't directly related to what we're talking about here, since--as we've just been told--Google's quality evaluators aren't doling out penalties or manipulating rankings.

reseller

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 9:07 pm on Jul 10, 2007 (gmt 0)

Folks!

This whole human-role-in-google-search thing is a farse, IMO.

Google has been preaching algo driven serps for years. Then few semi-human-edited and fully-human-edited-search-engines emerged.

Mighty GOOG got worried, and thought; lets jump on the human-edited-search-engines wagon.

And suddenly we see the kind Googlers busy telling the world; Ooooops...we forgot to tell ya that Google in fact has always been a human-edited-search-engine.

The remaining question to be answered by GOOG is: does the quality of human-edited-serps better than that of algo-driven-serps?

Good night and God bless.

glenster73

5+ Year Member



 
Msg#: 3390143 posted 10:24 pm on Jul 10, 2007 (gmt 0)

10,000 silent OompaLoompa's perhaps?

Nobody ever goes in, nobody ever comes out... :-)

hyperkik

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3390143 posted 10:29 pm on Jul 10, 2007 (gmt 0)

I know a woman who works as an independent contractor for Google, in search quality. About the only thing she'll say about the work is that she signed a non-disclosure agreement.

the_nerd

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3390143 posted 10:44 pm on Jul 10, 2007 (gmt 0)

correct me if I'm wrong,

but 3 hors daily @ 15$ * 200 days * 10000 people = 90 million dollars per year - that's small change for a giant like Google, even if you add 50% on top of that for recruiting services and the like.

europeforvisitors



 
Msg#: 3390143 posted 10:47 pm on Jul 10, 2007 (gmt 0)

This whole human-role-in-google-search thing is a farse, IMO. Google has been preaching algo driven serps for years.

Search engines don't design themselves. Humans provide basic parameters and supply input, whether directly or via examples of relevant and irrelevant or good and bad pages for a "black box." Do a search for GoogleGuy's comments on how human evaluators are used, and re-read what maximillianos a few hours ago:

"We were told our data would be used for research purposes and would not directly affect the search results."

It's also worth noting that PageRank, which was one of Google's founding principles, requires human input. (Humans, not algorithms, create the links that serve as the "votes" on which PageRank is based.)

Swanson

10+ Year Member



 
Msg#: 3390143 posted 11:13 pm on Jul 10, 2007 (gmt 0)

Is it me or is this suggestion absolutely ridiculous in the extreme!

To base your whole business model on algorithmic technology which was the whole reason people used you in the first place - lets be honest anything apart from search has been a huge, huge, huge (revenue) failure for Google. And then employ 10k individuals to "validate" and provide data to filter the algorithm.

What the hell are they doing then, the results have not improved - they have not removed spam, more legitimised it through mass ebay, amazon, wikipedia, shopping comparison, holding domains, mfa - etc. etc. We all know that if you are ebay then the rules are different than a small site that has missing in action for months or even years - I know that because I have been on the other side of it where I got to speak (on behalf of my rather large company in my capacity of technical director) to Google where they advised on how to correct indexing and ranking problems and within hours it was sorted (we are talking millions of pages).

What do these 10k people do - try various keyphrases to check the results? Try bloody acting on what webmasters in this forum have been talking about for years - 301s, URL hijacking, shallow crawling, incorrect duplicate penalties etc etc. And that is for free!

It is lucky I now treat Google with the contempt it deserves and now think absolutely nothing about it other than to make money out of their search results (that is my job as a webmaster) - I once gave a damn about ethics and "growth", "building", "quality", "longevity" and "trust", now it is so crazily volatile that I have to give a damn about me. I can't now call up Google to ask them to sort my indexing and ranking anymore as I am on my own now - just like everyone else who is not part of a huge site delivering millions of dollars to Google.

europeforvisitors



 
Msg#: 3390143 posted 12:30 am on Jul 11, 2007 (gmt 0)

lets be honest anything apart from search has been a huge, huge, huge (revenue) failure for Google.

That sounds like a topic for the GOOG forum, not the Google Search News forum, but in any case, the statement is completely wrong, as one can see by reading any Google earnings report.

This 106 message thread spans 4 pages: < < 106 ( 1 [2] 3 4 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved