Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Google's Human Evaluators - 10,000 of them?

         

tedster

3:06 am on Jul 10, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



We've known for a while that Google uses human evaluation in evolving their algorithm. They've even got a patent for integrating the humans and the algorithm [webmasterworld.com]. On 2007-06-23, Google held a Scalability Conference in Seattle. Here's an interesting tidbit from a Q&A session with Marissa Mayer:

Q: How do they tell if they have bad results?
A: ...they have 10,000 human evaluators who are always manually checking the relevance of various results.

Article by attendee Dare Obasanjo [25hoursaday.com]

Next time we're trying to figure out some odd change in the SERPs, we might do well do remember this human factor, eh?

maximillianos

6:33 pm on Jul 10, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Did you get the job at that time through a similar job announcement like this one?

reseller - Yes, that look pretty much exactly like the job ad I responded to.

BillyS

6:40 pm on Jul 10, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>>Google employs over 10,000 full time people and has temporary employees as well. I would consider every one of them a "human evaluator" that has an impact on search results.

They don't want to listen to us kkobashi. I think you, Brett, and I are all on the same page.

reseller

6:49 pm on Jul 10, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



maximillianos

Thanks for feedback. Much appreciated.

But it seems they weren't interested at all whether the Raters themselves are webmasters owning for example business sites or are part of affiliate programs. I mean Google didn't care about the objectivity (fairness) of the Raters.

maximillianos

6:57 pm on Jul 10, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



But it seems they weren't interested at all whether the Raters themselves are webmasters owning for example business sites or are part of affiliate programs. I mean Google didn't care about the objectivity (fairness) of the Raters.

I did have to jump through hoops to get the job since I was a webmaster. I had to get some sort of special approval where they reviewed my accounts with G and my websites, etc. From that info, I do believe they filtered the type of sites I could/would be rating. Not positive about that...

maximillianos

6:58 pm on Jul 10, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



BTW - the interview/testing process is a pain. It takes about 10 hours of your time... but you do get paid for it.

They are very thorough.

reseller

6:59 pm on Jul 10, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



BillyS

They don't want to listen to us kkobashi. I think you, Brett, and I are all on the same page.

The problems with your argument are:

- We are not sure whether Marissa Mayer was referring to Google's 10.000 employees.

- Taking into account the poor quality of Google's serps in specific sectors until recently. And taking into account that 10.000 highly educated Google employees were doing the evaluation job. I wouldn't consider that as a good PR for the folks at the plex.

Better to balme it on the $10 raters ;-)

kkobashi

7:07 pm on Jul 10, 2007 (gmt 0)



BillyS: Yeah and I'm sure all of those 10,000 outsourced India temps know how to read Chinese websites too :o)

BenjArriola

7:39 pm on Jul 10, 2007 (gmt 0)

10+ Year Member



I see the side of this where people may feel misjudged on why a site ranks low or high, banned or not. But I also see the side where there can be some justification of having borderline techniques for Ajax and Flash sites doing image replacement, flash replacement, essentially are simply hidden text but not made to stuff with keywords. Having some human check it out can see if the purpose is really a bad purpose or not.

ogletree

7:44 pm on Jul 10, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



They also hire humans to read blogs and forums to see what webmasters are doing and talking about. They work in Seattle.

maximillianos

7:54 pm on Jul 10, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



They also hire humans to read blogs and forums to see what webmasters are doing and talking about. They work in Seattle.

They are watching us right now! ;-)

wanna_learn

8:12 pm on Jul 10, 2007 (gmt 0)

10+ Year Member



Being living in India and having several friends in Hydrabad working for big IT companies, I havent ever heard somebody working as an serach evaluator or whatever with Google... part time or full time.

Do they impose so harsh non disclosure policy?

There have to be many ex-evaluators who write blogs or post on Forums ;-)

rarethings55

8:34 pm on Jul 10, 2007 (gmt 0)

10+ Year Member



"In order to storm Rohan we would need thousands."

"tens of thousands".

"but my lord, there is no such army..."

Cut to the news.

skibum

8:40 pm on Jul 10, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



According to the ultimate Google authority (Wiki), they had 12,238 full time employees as of March 31, 2007 so virtually all of them would have to be pt/temp and if they are how could you count on them for QC?

Wasn't not doing hand manipulations some reason they put out in court when someone to dogde a lawsuit when somebody lost their rankings before? Maybe I've got that wrong...

europeforvisitors

8:52 pm on Jul 10, 2007 (gmt 0)



BTW - the interview/testing process is a pain. It takes about 10 hours of your time... but you do get paid for it.

They are very thorough.

Sounds like they aren't stupid, either.

Wasn't not doing hand manipulations some reason they put out in court when someone to dogde a lawsuit when somebody lost their rankings before? Maybe I've got that wrong...

You're probably thinking of the SearchKing case (which SearchKing lost), but I don't believe Google has ever claimed that it doesn't do manual penalties. Indeed, the court decision reaffirmed Google's first-amendment right to adjust rankings or remove pages from its search results. Still, that isn't directly related to what we're talking about here, since--as we've just been told--Google's quality evaluators aren't doling out penalties or manipulating rankings.

reseller

9:07 pm on Jul 10, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Folks!

This whole human-role-in-google-search thing is a farse, IMO.

Google has been preaching algo driven serps for years. Then few semi-human-edited and fully-human-edited-search-engines emerged.

Mighty GOOG got worried, and thought; lets jump on the human-edited-search-engines wagon.

And suddenly we see the kind Googlers busy telling the world; Ooooops...we forgot to tell ya that Google in fact has always been a human-edited-search-engine.

The remaining question to be answered by GOOG is: does the quality of human-edited-serps better than that of algo-driven-serps?

Good night and God bless.

glenster73

10:24 pm on Jul 10, 2007 (gmt 0)

10+ Year Member



10,000 silent OompaLoompa's perhaps?

Nobody ever goes in, nobody ever comes out... :-)

hyperkik

10:29 pm on Jul 10, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I know a woman who works as an independent contractor for Google, in search quality. About the only thing she'll say about the work is that she signed a non-disclosure agreement.

the_nerd

10:44 pm on Jul 10, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



correct me if I'm wrong,

but 3 hors daily @ 15$ * 200 days * 10000 people = 90 million dollars per year - that's small change for a giant like Google, even if you add 50% on top of that for recruiting services and the like.

europeforvisitors

10:47 pm on Jul 10, 2007 (gmt 0)



This whole human-role-in-google-search thing is a farse, IMO. Google has been preaching algo driven serps for years.

Search engines don't design themselves. Humans provide basic parameters and supply input, whether directly or via examples of relevant and irrelevant or good and bad pages for a "black box." Do a search for GoogleGuy's comments on how human evaluators are used, and re-read what maximillianos a few hours ago:

"We were told our data would be used for research purposes and would not directly affect the search results."

It's also worth noting that PageRank, which was one of Google's founding principles, requires human input. (Humans, not algorithms, create the links that serve as the "votes" on which PageRank is based.)

Swanson

11:13 pm on Jul 10, 2007 (gmt 0)

10+ Year Member



Is it me or is this suggestion absolutely ridiculous in the extreme!

To base your whole business model on algorithmic technology which was the whole reason people used you in the first place - lets be honest anything apart from search has been a huge, huge, huge (revenue) failure for Google. And then employ 10k individuals to "validate" and provide data to filter the algorithm.

What the hell are they doing then, the results have not improved - they have not removed spam, more legitimised it through mass ebay, amazon, wikipedia, shopping comparison, holding domains, mfa - etc. etc. We all know that if you are ebay then the rules are different than a small site that has missing in action for months or even years - I know that because I have been on the other side of it where I got to speak (on behalf of my rather large company in my capacity of technical director) to Google where they advised on how to correct indexing and ranking problems and within hours it was sorted (we are talking millions of pages).

What do these 10k people do - try various keyphrases to check the results? Try bloody acting on what webmasters in this forum have been talking about for years - 301s, URL hijacking, shallow crawling, incorrect duplicate penalties etc etc. And that is for free!

It is lucky I now treat Google with the contempt it deserves and now think absolutely nothing about it other than to make money out of their search results (that is my job as a webmaster) - I once gave a damn about ethics and "growth", "building", "quality", "longevity" and "trust", now it is so crazily volatile that I have to give a damn about me. I can't now call up Google to ask them to sort my indexing and ranking anymore as I am on my own now - just like everyone else who is not part of a huge site delivering millions of dollars to Google.

This 106 message thread spans 6 pages: 106