Welcome to WebmasterWorld Guest from 54.147.134.218

Message Too Old, No Replies

Google's Human Evaluators - 10,000 of them?

     
3:06 am on Jul 10, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


We've known for a while that Google uses human evaluation in evolving their algorithm. They've even got a patent for integrating the humans and the algorithm [webmasterworld.com]. On 2007-06-23, Google held a Scalability Conference in Seattle. Here's an interesting tidbit from a Q&A session with Marissa Mayer:

Q: How do they tell if they have bad results?
A: ...they have 10,000 human evaluators who are always manually checking the relevance of various results.

Article by attendee Dare Obasanjo [25hoursaday.com]

Next time we're trying to figure out some odd change in the SERPs, we might do well do remember this human factor, eh?

12:30 am on July 11, 2007 (gmt 0)

Senior Member

joined:Oct 27, 2001
posts:10210
votes: 0


lets be honest anything apart from search has been a huge, huge, huge (revenue) failure for Google.

That sounds like a topic for the GOOG forum, not the Google Search News forum, but in any case, the statement is completely wrong, as one can see by reading any Google earnings report.

12:58 am on July 11, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member whitey is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 28, 2005
posts:3055
votes: 16


I think this demonstrates that Google has the capacity through it's management systems , user data, search alogorithms and human reach [ users and employees ] to control any search result set.

It's just a matter of improving organisation.

So we need to believe that G will get progressively more serious about content and linking technique control. When this is complete few sites will remain in their positions , by links alone and content alone. What will evolve are techniques relying solely on background data comprising such comparative things as adherence to guidelines and user activity measurements of quality.

These types of things are the new voting paramaters for a website. I think links will become a thing of the past, and I recall some recent posts indicating these thoughts to.

This does mean that G will move to a new level of complete dominance IMO

1:51 am on July 11, 2007 (gmt 0)

Senior Member

joined:Oct 27, 2001
posts:10210
votes: 0


These types of things are the new voting paramaters for a website. I think links will become a thing of the past, and I recall some recent posts indicating these thoughts to.

I don't think links will become a thing of the past with Google; they just won't be as powerful by themselves as they used to be, and Google will continue to improve its ability to distinguish between valuable and meaningless links.

3:00 am on July 11, 2007 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member ogletree is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 14, 2003
posts:4249
votes: 16


I really don't believe that they have 10k people that just work on SERPS. Maybe each employee is supposed to look at a few SERPS every day and send a report. Maybe they are talking about spam reports from webmaster central. If there really were 10k people doing this full time there would be some large facility and there would be job postings and there would be people that were fired or left the job that talked about it.
3:28 am on July 11, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 18, 2005
posts:61
votes: 0


If it is true...

I think those human evaluators have a say on the ranking of the top few hundred sites..

4:17 am on July 11, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 31, 2002
posts:93
votes: 0


>>I think those human evaluators have a say on the ranking of the top few hundred sites..

You could pretty well kill the bias rate by recycling the same SERP to a variety of reviewers, wisdom of crowd type of thing.

6:13 am on July 11, 2007 (gmt 0)

Junior Member

5+ Year Member

joined:June 19, 2007
posts:53
votes: 0


basically I almost applied for this job, and I have a friend that does have this job. mostly foreign language stuff that I saw.

they pay around $12 - $15 an hour, work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites.

7:01 am on July 11, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


hippypink

Thanks for feedback.

they pay around $12 - $15 an hour, work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites.

And that might explain the poor quality of Google's serps that have been mentioned on different threads on WebmasterWorld for the latest years.

We know that multi billion Google could afford to spend more to improve its serps quality but they haven't unfortunately.

IMO, better algos driven serps than the $12 - $15 evaluators you and another friends have mentioned on this thread.

Talking about the human-role-in-Google :-)

7:06 am on July 11, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member

joined:Apr 1, 2006
posts:1536
votes: 0


Just for your info. USD 15 / Hr is top money in India. A senior well qualified executive or a Project manager in a top IT co. would earn something like USD 4000 / Month.
7:16 am on July 11, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


Green_Grass

I'm sure that $12 - $15 is top money also in Mogadishu in Somalia for "work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites."

Oh well...you get what you pay for :-)

7:52 am on July 11, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


Its also amazing to see a Godfather of algorithms [en.wikipedia.org] VP of Engineering at Google Udi Manber [google.com], just like being forced to talk about the role of humans in algorithms :-)
Wonder whether Udi is aware of the the role of the $10 evaluators in his algorithms too :-)

For me the role of human in Google is better explained in the great efforts of kind Googlers like Matt Cutts, Adam Lasnik and Vanessa Fox to communicate with the webmasters communities on forums and during conferences. Power to Matt, Adam and Vanessa!

10:39 am on July 11, 2007 (gmt 0)

Junior Member

5+ Year Member

joined:Mar 7, 2006
posts:170
votes: 0


It was mentioned earier about the Google Toolbar, could it simply be that they have 10,000 active users who use the vote buttons? Or would the numbers be far larger if this was the case.
11:56 am on July 11, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


scotland

It was mentioned earier about the Google Toolbar, could it simply be that they have 10,000 active users who use the vote buttons?

Possibly the human evaluators work is related to Random Query Evaluation [google.com].

4:04 pm on July 11, 2007 (gmt 0)

New User

5+ Year Member

joined:Apr 18, 2007
posts:17
votes: 0


The "revelation" that they have so many human evaluators is refreshing and expected to me. Google is, in a Matrix or Terminator movie sort of way, about machines, computers, algorithms, etc replacing humans. I've always said that search engines aren't about search anyway, they are about advertising. They perfected algorithms not to give me a "free" search engine but to keep them from embarrassingly putting a Victoria Secret ad on my Cub Scout website.

Google needs to maintain its aura of supreme ad context management to maintain its market position. To rely on human editors, after all, is an admission that Google is just another Yahoo or DMOZ with a different story line. They wouldn't want that now, would they?

5:18 pm on July 11, 2007 (gmt 0)

Senior Member

joined:Oct 27, 2001
posts:10210
votes: 0


To rely on human editors, after all, is an admission that Google is just another Yahoo or DMOZ with a different story line.

Fortunately, they'll never have to make such an admission, since they don't "rely on human editors."

8:32 pm on July 11, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


From what I read, it seems Google is moving from algorithms-driven-search engine to the direction of semi-human-edited-search engine. As such its going to rely on human editors or human evaluators. I.e those 10.000 evaluators are going to affect Google's serps directly!

Here is [mattcutts.com] what some Googlers said:

- Marissa Mayer : “Up to today we have relied on automation, but I believe the future will be a blend of both, combing the scale of automation and human intelligence.”

- Larry Page: "Larry says search is finding content… and that Wikipedia found a better way to organize information. he seems to like the model of using humans and process and machines."

- Matt Cutts: "So I think too many people get hung up on “Google having algorithms.” They miss the larger picture, which (to me) is to pursue approaches that are scalable and robust, even if that implies a human side. There’s nothing inherently wrong with using contributions from people–you just have to bear in mind the limitations of that data."

- Udi Manber said [stonetemple.com]: "If somebody says they don’t like a particular search result that’s a signal. So, we’ve been using that for a long time, and we are working on new ways of using it."

Its therefore of importance to know exactly which 10.000 human evaluators Marissa Mayer was referring to in her session with Dare Obasanjo. Was she referring to Google internal employees or the $10 - $15 temp evaluators worldwide?

[edited by: reseller at 8:56 pm (utc) on July 11, 2007]

8:56 pm on July 11, 2007 (gmt 0)

Senior Member

joined:Oct 27, 2001
posts:10210
votes: 0


From what I read, it seems Google is moving from algorithms-driven-search engine to the direction of semi-human-edited-search engine.

You're reading your own preconceptions into what was actually said. There's no way that a "semi-human-edited" search engine would be "scalable," to use a popular Google term. Where human input can be valuable is in providing cues and clues for writing and/or training the software that does the work. (And, of course, Google has been doing that all along; PageRank is based on human input, and it's highly unlikely that Larry Page and Sergey Brin let their original algorithm write itself back in the late 1990s.)

ADDENDUM: The proof is in the pudding, and having paid evaluators to help identify spam, distinguish between good and bad results, etc. should improve Google's search results over the long term--regardless of where they're being recruited, how much they're being paid, and whether their input is being used to train software, to facilitate research, or (less likely) to simply whack pages that don't deserve to be in Google's SERPs.

10:19 pm on July 11, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


EFV

..and having paid evaluators to help identify spam, distinguish between good and bad results, etc. should improve Google's search results over the long term--regardless of where they're being recruited, how much they're being paid,

Its not only how much they are being paid which worry me. Its a statement like this one [webmasterworld.com] too:

"they pay around $12 - $15 an hour, work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites."

Would any site owner and webmaster feel comfortable knowing that such kind of $12 - $15 evaluators might affect directly or indirectly which site to stay and which site to be deindexed on Google index?

10:30 pm on July 11, 2007 (gmt 0)

Senior Member

joined:Oct 27, 2001
posts:10210
votes: 0


Would any site owner and webmaster feel comfortable knowing that such kind of $12 - $15 evaluators might affect directly or indirectly which site to stay and which site to be deindexed on Google index?

I imagine that depends, to a great extent, on whether the site owner or Webmaster has anything to hide. :-)

11:16 pm on July 11, 2007 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member ogletree is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 14, 2003
posts:4249
votes: 16


The problem with this is a bunch of people will blame any problems they have to human evaluators. Just like they blame penalty's or any other theory on here. Most of the time it just comes down to quality links and content or obvious spam tactics.
11:51 pm on July 11, 2007 (gmt 0)

New User

joined:Mar 31, 2007
posts:33
votes: 0


That patent isn't about using human evaluators for rating. It's about developing a classification system for search queries ("query themes") and pages: "For the query theme "sites that provide free software downloads," the rule may be the requirement that the query contain the word "free" and "download." For the query theme "sites that help in finding an accommodation," the rule may be the requirement that the query contain one of the words: {"accommodation," "lodging," "hotels," . . . } and also that the query contains the name of a place (e.g., by matching one of a list of place names)." That's a scheme to detect special cases and route them off to a special search system. Yahoo was first to do that, with about 30 special cases (celebrities, sports, etc.), and Google is now moving in that direction.

There are patents for schemes which explicitly use user feedback to adjust ratings, and most of those patents are from Neelakantan Sundaresan at IBM Almaden Research, not from Google.

The fundamental trouble with using user feedback to drive search is that it's too easy to spam. Unless the number of raters per thing rated is high, as for movies, or you have to do an actual transaction with real money to submit a rating, as with eBay, most ratings will come from involved parties. Which does not help.

12:37 am on July 12, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 28, 2003
posts: 167
votes: 0


I was one of the evaluators for a while. There's no way they have 10,000 - she must've been cleverly making it sound that way, whereas the 10,000 figure is for all Google employees.

They give multiple evaluators the same work, so that they get a consensus answer. Very basic "is this page spam or not" work. The purpose is to check examples of spam, not all spam. 10,000 evaluators would imply that they are making dozens of tweaks to the algorithm every day, and have 500 staff doing the tweaks... Google's search algorithm is too important to have that many people working on it at once.

I'd guess the real figures to be more like 20 working on the algorithm, and 100-300 evaluators.

1:42 am on July 12, 2007 (gmt 0)

Senior Member

joined:Oct 27, 2001
posts:10210
votes: 0


The problem with this is a bunch of people will blame any problems they have to human evaluators.

I don't think Google would regard that as a problem. They must be pretty thick-skinned by now. :-)

7:42 am on July 12, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


robertskelton

There's no way they have 10,000 - she must've been cleverly making it sound that way, whereas the 10,000 figure is for all Google employees.

Who knows what are those 10.000 are (:(

12:21 pm on July 12, 2007 (gmt 0)

New User

10+ Year Member

joined:Dec 2, 2004
posts:30
votes: 0


Let's just ask them what
AASearch
AAPub
AAShop
means ;)
12:38 pm on July 12, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 6, 2005
posts:1678
votes: 71


robertskelton

Btw, did Google do anything to secure the objectivity (fairness) of the evaluators work? any special requirements?

12:42 pm on July 12, 2007 (gmt 0)

New User

10+ Year Member

joined:Dec 2, 2004
posts:30
votes: 0


or at least secure the tool :))
1:37 pm on July 12, 2007 (gmt 0)

New User

5+ Year Member

joined:May 14, 2007
posts:13
votes: 0


I've been saying this for years.

The bottom line is that 90% of the SEO tricks that we all learned in 1999 are worthless today. You cannot fool Google into thinking that your site deserves to be #1 in the results through some kind of trickery. Maybe you can temporarily, but in the end, you better have a site that looks good and works good and that people like. If you don't, all the meta-tag tweaking in the world isn't going to make a difference.

For years my company has been designing sites, and after a year or two online, the customer comes back to us saying "This is incredible! Google LOVES us!" Now we know why. Because the sites we design are good, they look good, they work good, and when the human evaluator comes to look at the site for even a minute or two, they give it the thumbs up.

It's been so obvious that they do this, just by watching Google's behavior. People said I was crazy, they said that there are too many websites out there to do this, they said that Google has publically stated that they seek to completely automate their process. But I don't care what anybody says--I care about what their software does. And their software has been screaming of human interference loud and clear ever since the day they went online.

That's why I feel completely justified in screaming "HAH! I TOLD YOU SO!" at the top of my lungs right now.

Cheers,
Bolotomus

2:58 pm on July 12, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 7, 2005
posts:137
votes: 0


Could it be that when a reinclusion request is filed, several human evaluators give their opinions?
3:25 pm on July 12, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member hobbs is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 19, 2004
posts:3054
votes: 3


Scan your logs for:

google.com/evaluation/search/rating/task-edit?task=

Get the IP, and find all hits from that IP in your logs:

Here is what I am seeing from the same (non Google) IP:

- Page1 is requested with a direct hit, no referrer (He is browsing the web)
- 43 min. later same page is hit giving the evaluation as referrer (he decided to evaluate my page)
- 30 seconds later the same page is hit, no referrer (He hit an external link on the page and then the back button)
- 2.5 Hours later same page is hit giving the evaluation as referrer (I have no idea why)
- He bookmarks the page (Plans to visit again)

And this is a very minor trivial page on my site, the same IP does not surf to other links on the page, does not try to find out more about my site or what it does, just evaluating the page independently from anything else, I have also looked up everyone else that requested the page since and it is not visited by any Google IP or another evaluation referrer.

[edited by: Hobbs at 3:33 pm (utc) on July 12, 2007]

This 106 message thread spans 4 pages: 106
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members