Welcome to WebmasterWorld Guest from 54.145.166.96

Message Too Old, No Replies

Google's Human Evaluators - 10,000 of them?

   
3:06 am on Jul 10, 2007 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



We've known for a while that Google uses human evaluation in evolving their algorithm. They've even got a patent for integrating the humans and the algorithm [webmasterworld.com]. On 2007-06-23, Google held a Scalability Conference in Seattle. Here's an interesting tidbit from a Q&A session with Marissa Mayer:

Q: How do they tell if they have bad results?
A: ...they have 10,000 human evaluators who are always manually checking the relevance of various results.

Article by attendee Dare Obasanjo [25hoursaday.com]

Next time we're trying to figure out some odd change in the SERPs, we might do well do remember this human factor, eh?

12:30 am on Jul 11, 2007 (gmt 0)



lets be honest anything apart from search has been a huge, huge, huge (revenue) failure for Google.

That sounds like a topic for the GOOG forum, not the Google Search News forum, but in any case, the statement is completely wrong, as one can see by reading any Google earnings report.

12:58 am on Jul 11, 2007 (gmt 0)

WebmasterWorld Senior Member whitey is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I think this demonstrates that Google has the capacity through it's management systems , user data, search alogorithms and human reach [ users and employees ] to control any search result set.

It's just a matter of improving organisation.

So we need to believe that G will get progressively more serious about content and linking technique control. When this is complete few sites will remain in their positions , by links alone and content alone. What will evolve are techniques relying solely on background data comprising such comparative things as adherence to guidelines and user activity measurements of quality.

These types of things are the new voting paramaters for a website. I think links will become a thing of the past, and I recall some recent posts indicating these thoughts to.

This does mean that G will move to a new level of complete dominance IMO

1:51 am on Jul 11, 2007 (gmt 0)



These types of things are the new voting paramaters for a website. I think links will become a thing of the past, and I recall some recent posts indicating these thoughts to.

I don't think links will become a thing of the past with Google; they just won't be as powerful by themselves as they used to be, and Google will continue to improve its ability to distinguish between valuable and meaningless links.

3:00 am on Jul 11, 2007 (gmt 0)

WebmasterWorld Senior Member ogletree is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I really don't believe that they have 10k people that just work on SERPS. Maybe each employee is supposed to look at a few SERPS every day and send a report. Maybe they are talking about spam reports from webmaster central. If there really were 10k people doing this full time there would be some large facility and there would be job postings and there would be people that were fired or left the job that talked about it.
3:28 am on Jul 11, 2007 (gmt 0)

10+ Year Member



If it is true...

I think those human evaluators have a say on the ranking of the top few hundred sites..

4:17 am on Jul 11, 2007 (gmt 0)

10+ Year Member



>>I think those human evaluators have a say on the ranking of the top few hundred sites..

You could pretty well kill the bias rate by recycling the same SERP to a variety of reviewers, wisdom of crowd type of thing.

6:13 am on Jul 11, 2007 (gmt 0)

5+ Year Member



basically I almost applied for this job, and I have a friend that does have this job. mostly foreign language stuff that I saw.

they pay around $12 - $15 an hour, work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites.

7:01 am on Jul 11, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



hippypink

Thanks for feedback.

they pay around $12 - $15 an hour, work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites.

And that might explain the poor quality of Google's serps that have been mentioned on different threads on WebmasterWorld for the latest years.

We know that multi billion Google could afford to spend more to improve its serps quality but they haven't unfortunately.

IMO, better algos driven serps than the $12 - $15 evaluators you and another friends have mentioned on this thread.

Talking about the human-role-in-Google :-)

7:06 am on Jul 11, 2007 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Just for your info. USD 15 / Hr is top money in India. A senior well qualified executive or a Project manager in a top IT co. would earn something like USD 4000 / Month.
7:16 am on Jul 11, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Green_Grass

I'm sure that $12 - $15 is top money also in Mogadishu in Somalia for "work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites."

Oh well...you get what you pay for :-)

7:52 am on Jul 11, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Its also amazing to see a Godfather of algorithms [en.wikipedia.org] VP of Engineering at Google Udi Manber [google.com], just like being forced to talk about the role of humans in algorithms :-)
Wonder whether Udi is aware of the the role of the $10 evaluators in his algorithms too :-)

For me the role of human in Google is better explained in the great efforts of kind Googlers like Matt Cutts, Adam Lasnik and Vanessa Fox to communicate with the webmasters communities on forums and during conferences. Power to Matt, Adam and Vanessa!

10:39 am on Jul 11, 2007 (gmt 0)

5+ Year Member



It was mentioned earier about the Google Toolbar, could it simply be that they have 10,000 active users who use the vote buttons? Or would the numbers be far larger if this was the case.
11:56 am on Jul 11, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



scotland

It was mentioned earier about the Google Toolbar, could it simply be that they have 10,000 active users who use the vote buttons?

Possibly the human evaluators work is related to Random Query Evaluation [google.com].

4:04 pm on Jul 11, 2007 (gmt 0)

5+ Year Member



The "revelation" that they have so many human evaluators is refreshing and expected to me. Google is, in a Matrix or Terminator movie sort of way, about machines, computers, algorithms, etc replacing humans. I've always said that search engines aren't about search anyway, they are about advertising. They perfected algorithms not to give me a "free" search engine but to keep them from embarrassingly putting a Victoria Secret ad on my Cub Scout website.

Google needs to maintain its aura of supreme ad context management to maintain its market position. To rely on human editors, after all, is an admission that Google is just another Yahoo or DMOZ with a different story line. They wouldn't want that now, would they?

5:18 pm on Jul 11, 2007 (gmt 0)



To rely on human editors, after all, is an admission that Google is just another Yahoo or DMOZ with a different story line.

Fortunately, they'll never have to make such an admission, since they don't "rely on human editors."

8:32 pm on Jul 11, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



From what I read, it seems Google is moving from algorithms-driven-search engine to the direction of semi-human-edited-search engine. As such its going to rely on human editors or human evaluators. I.e those 10.000 evaluators are going to affect Google's serps directly!

Here is [mattcutts.com] what some Googlers said:

- Marissa Mayer : “Up to today we have relied on automation, but I believe the future will be a blend of both, combing the scale of automation and human intelligence.”

- Larry Page: "Larry says search is finding content… and that Wikipedia found a better way to organize information. he seems to like the model of using humans and process and machines."

- Matt Cutts: "So I think too many people get hung up on “Google having algorithms.” They miss the larger picture, which (to me) is to pursue approaches that are scalable and robust, even if that implies a human side. There’s nothing inherently wrong with using contributions from people–you just have to bear in mind the limitations of that data."

- Udi Manber said [stonetemple.com]: "If somebody says they don’t like a particular search result that’s a signal. So, we’ve been using that for a long time, and we are working on new ways of using it."

Its therefore of importance to know exactly which 10.000 human evaluators Marissa Mayer was referring to in her session with Dare Obasanjo. Was she referring to Google internal employees or the $10 - $15 temp evaluators worldwide?

[edited by: reseller at 8:56 pm (utc) on July 11, 2007]

8:56 pm on Jul 11, 2007 (gmt 0)



From what I read, it seems Google is moving from algorithms-driven-search engine to the direction of semi-human-edited-search engine.

You're reading your own preconceptions into what was actually said. There's no way that a "semi-human-edited" search engine would be "scalable," to use a popular Google term. Where human input can be valuable is in providing cues and clues for writing and/or training the software that does the work. (And, of course, Google has been doing that all along; PageRank is based on human input, and it's highly unlikely that Larry Page and Sergey Brin let their original algorithm write itself back in the late 1990s.)

ADDENDUM: The proof is in the pudding, and having paid evaluators to help identify spam, distinguish between good and bad results, etc. should improve Google's search results over the long term--regardless of where they're being recruited, how much they're being paid, and whether their input is being used to train software, to facilitate research, or (less likely) to simply whack pages that don't deserve to be in Google's SERPs.

10:19 pm on Jul 11, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



EFV

..and having paid evaluators to help identify spam, distinguish between good and bad results, etc. should improve Google's search results over the long term--regardless of where they're being recruited, how much they're being paid,

Its not only how much they are being paid which worry me. Its a statement like this one [webmasterworld.com] too:

"they pay around $12 - $15 an hour, work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites."

Would any site owner and webmaster feel comfortable knowing that such kind of $12 - $15 evaluators might affect directly or indirectly which site to stay and which site to be deindexed on Google index?

10:30 pm on Jul 11, 2007 (gmt 0)



Would any site owner and webmaster feel comfortable knowing that such kind of $12 - $15 evaluators might affect directly or indirectly which site to stay and which site to be deindexed on Google index?

I imagine that depends, to a great extent, on whether the site owner or Webmaster has anything to hide. :-)

11:16 pm on Jul 11, 2007 (gmt 0)

WebmasterWorld Senior Member ogletree is a WebmasterWorld Top Contributor of All Time 10+ Year Member



The problem with this is a bunch of people will blame any problems they have to human evaluators. Just like they blame penalty's or any other theory on here. Most of the time it just comes down to quality links and content or obvious spam tactics.
11:51 pm on Jul 11, 2007 (gmt 0)



That patent isn't about using human evaluators for rating. It's about developing a classification system for search queries ("query themes") and pages: "For the query theme "sites that provide free software downloads," the rule may be the requirement that the query contain the word "free" and "download." For the query theme "sites that help in finding an accommodation," the rule may be the requirement that the query contain one of the words: {"accommodation," "lodging," "hotels," . . . } and also that the query contains the name of a place (e.g., by matching one of a list of place names)." That's a scheme to detect special cases and route them off to a special search system. Yahoo was first to do that, with about 30 special cases (celebrities, sports, etc.), and Google is now moving in that direction.

There are patents for schemes which explicitly use user feedback to adjust ratings, and most of those patents are from Neelakantan Sundaresan at IBM Almaden Research, not from Google.

The fundamental trouble with using user feedback to drive search is that it's too easy to spam. Unless the number of raters per thing rated is high, as for movies, or you have to do an actual transaction with real money to submit a rating, as with eBay, most ratings will come from involved parties. Which does not help.

12:37 am on Jul 12, 2007 (gmt 0)

10+ Year Member



I was one of the evaluators for a while. There's no way they have 10,000 - she must've been cleverly making it sound that way, whereas the 10,000 figure is for all Google employees.

They give multiple evaluators the same work, so that they get a consensus answer. Very basic "is this page spam or not" work. The purpose is to check examples of spam, not all spam. 10,000 evaluators would imply that they are making dozens of tweaks to the algorithm every day, and have 500 staff doing the tweaks... Google's search algorithm is too important to have that many people working on it at once.

I'd guess the real figures to be more like 20 working on the algorithm, and 100-300 evaluators.

1:42 am on Jul 12, 2007 (gmt 0)



The problem with this is a bunch of people will blame any problems they have to human evaluators.

I don't think Google would regard that as a problem. They must be pretty thick-skinned by now. :-)

7:42 am on Jul 12, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



robertskelton

There's no way they have 10,000 - she must've been cleverly making it sound that way, whereas the 10,000 figure is for all Google employees.

Who knows what are those 10.000 are (:(

12:21 pm on Jul 12, 2007 (gmt 0)

10+ Year Member



Let's just ask them what
AASearch
AAPub
AAShop
means ;)
12:38 pm on Jul 12, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



robertskelton

Btw, did Google do anything to secure the objectivity (fairness) of the evaluators work? any special requirements?

12:42 pm on Jul 12, 2007 (gmt 0)

10+ Year Member



or at least secure the tool :))
1:37 pm on Jul 12, 2007 (gmt 0)

5+ Year Member



I've been saying this for years.

The bottom line is that 90% of the SEO tricks that we all learned in 1999 are worthless today. You cannot fool Google into thinking that your site deserves to be #1 in the results through some kind of trickery. Maybe you can temporarily, but in the end, you better have a site that looks good and works good and that people like. If you don't, all the meta-tag tweaking in the world isn't going to make a difference.

For years my company has been designing sites, and after a year or two online, the customer comes back to us saying "This is incredible! Google LOVES us!" Now we know why. Because the sites we design are good, they look good, they work good, and when the human evaluator comes to look at the site for even a minute or two, they give it the thumbs up.

It's been so obvious that they do this, just by watching Google's behavior. People said I was crazy, they said that there are too many websites out there to do this, they said that Google has publically stated that they seek to completely automate their process. But I don't care what anybody says--I care about what their software does. And their software has been screaming of human interference loud and clear ever since the day they went online.

That's why I feel completely justified in screaming "HAH! I TOLD YOU SO!" at the top of my lungs right now.

Cheers,
Bolotomus

2:58 pm on Jul 12, 2007 (gmt 0)

5+ Year Member



Could it be that when a reinclusion request is filed, several human evaluators give their opinions?
3:25 pm on Jul 12, 2007 (gmt 0)

WebmasterWorld Senior Member hobbs is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Scan your logs for:

google.com/evaluation/search/rating/task-edit?task=

Get the IP, and find all hits from that IP in your logs:

Here is what I am seeing from the same (non Google) IP:

- Page1 is requested with a direct hit, no referrer (He is browsing the web)
- 43 min. later same page is hit giving the evaluation as referrer (he decided to evaluate my page)
- 30 seconds later the same page is hit, no referrer (He hit an external link on the page and then the back button)
- 2.5 Hours later same page is hit giving the evaluation as referrer (I have no idea why)
- He bookmarks the page (Plans to visit again)

And this is a very minor trivial page on my site, the same IP does not surf to other links on the page, does not try to find out more about my site or what it does, just evaluating the page independently from anything else, I have also looked up everyone else that requested the page since and it is not visited by any Google IP or another evaluation referrer.

[edited by: Hobbs at 3:33 pm (utc) on July 12, 2007]

This 106 message thread spans 4 pages: 106
 

Featured Threads

Hot Threads This Week

Hot Threads This Month