homepage Welcome to WebmasterWorld Guest from 54.197.15.196
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

This 106 message thread spans 4 pages: < < 106 ( 1 2 [3] 4 > >     
Google's Human Evaluators - 10,000 of them?
tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3390143 posted 3:06 am on Jul 10, 2007 (gmt 0)

We've known for a while that Google uses human evaluation in evolving their algorithm. They've even got a patent for integrating the humans and the algorithm [webmasterworld.com]. On 2007-06-23, Google held a Scalability Conference in Seattle. Here's an interesting tidbit from a Q&A session with Marissa Mayer:

Q: How do they tell if they have bad results?
A: ...they have 10,000 human evaluators who are always manually checking the relevance of various results.

Article by attendee Dare Obasanjo [25hoursaday.com]

Next time we're trying to figure out some odd change in the SERPs, we might do well do remember this human factor, eh?

 

europeforvisitors



 
Msg#: 3390143 posted 12:30 am on Jul 11, 2007 (gmt 0)

lets be honest anything apart from search has been a huge, huge, huge (revenue) failure for Google.

That sounds like a topic for the GOOG forum, not the Google Search News forum, but in any case, the statement is completely wrong, as one can see by reading any Google earnings report.

Whitey

WebmasterWorld Senior Member whitey us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 3390143 posted 12:58 am on Jul 11, 2007 (gmt 0)

I think this demonstrates that Google has the capacity through it's management systems , user data, search alogorithms and human reach [ users and employees ] to control any search result set.

It's just a matter of improving organisation.

So we need to believe that G will get progressively more serious about content and linking technique control. When this is complete few sites will remain in their positions , by links alone and content alone. What will evolve are techniques relying solely on background data comprising such comparative things as adherence to guidelines and user activity measurements of quality.

These types of things are the new voting paramaters for a website. I think links will become a thing of the past, and I recall some recent posts indicating these thoughts to.

This does mean that G will move to a new level of complete dominance IMO

europeforvisitors



 
Msg#: 3390143 posted 1:51 am on Jul 11, 2007 (gmt 0)

These types of things are the new voting paramaters for a website. I think links will become a thing of the past, and I recall some recent posts indicating these thoughts to.

I don't think links will become a thing of the past with Google; they just won't be as powerful by themselves as they used to be, and Google will continue to improve its ability to distinguish between valuable and meaningless links.

ogletree

WebmasterWorld Senior Member ogletree us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3390143 posted 3:00 am on Jul 11, 2007 (gmt 0)

I really don't believe that they have 10k people that just work on SERPS. Maybe each employee is supposed to look at a few SERPS every day and send a report. Maybe they are talking about spam reports from webmaster central. If there really were 10k people doing this full time there would be some large facility and there would be job postings and there would be people that were fired or left the job that talked about it.

testy

5+ Year Member



 
Msg#: 3390143 posted 3:28 am on Jul 11, 2007 (gmt 0)

If it is true...

I think those human evaluators have a say on the ranking of the top few hundred sites..

unreviewed

10+ Year Member



 
Msg#: 3390143 posted 4:17 am on Jul 11, 2007 (gmt 0)

>>I think those human evaluators have a say on the ranking of the top few hundred sites..

You could pretty well kill the bias rate by recycling the same SERP to a variety of reviewers, wisdom of crowd type of thing.

hippypink

5+ Year Member



 
Msg#: 3390143 posted 6:13 am on Jul 11, 2007 (gmt 0)

basically I almost applied for this job, and I have a friend that does have this job. mostly foreign language stuff that I saw.

they pay around $12 - $15 an hour, work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites.

reseller

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 7:01 am on Jul 11, 2007 (gmt 0)

hippypink

Thanks for feedback.

they pay around $12 - $15 an hour, work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites.

And that might explain the poor quality of Google's serps that have been mentioned on different threads on WebmasterWorld for the latest years.

We know that multi billion Google could afford to spend more to improve its serps quality but they haven't unfortunately.

IMO, better algos driven serps than the $12 - $15 evaluators you and another friends have mentioned on this thread.

Talking about the human-role-in-Google :-)

Green_Grass

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 7:06 am on Jul 11, 2007 (gmt 0)

Just for your info. USD 15 / Hr is top money in India. A senior well qualified executive or a Project manager in a top IT co. would earn something like USD 4000 / Month.

reseller

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 7:16 am on Jul 11, 2007 (gmt 0)

Green_Grass

I'm sure that $12 - $15 is top money also in Mogadishu in Somalia for "work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites."

Oh well...you get what you pay for :-)

reseller

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 7:52 am on Jul 11, 2007 (gmt 0)

Its also amazing to see a Godfather of algorithms [en.wikipedia.org] VP of Engineering at Google Udi Manber [google.com], just like being forced to talk about the role of humans in algorithms :-)
Wonder whether Udi is aware of the the role of the $10 evaluators in his algorithms too :-)

For me the role of human in Google is better explained in the great efforts of kind Googlers like Matt Cutts, Adam Lasnik and Vanessa Fox to communicate with the webmasters communities on forums and during conferences. Power to Matt, Adam and Vanessa!

scotland

5+ Year Member



 
Msg#: 3390143 posted 10:39 am on Jul 11, 2007 (gmt 0)

It was mentioned earier about the Google Toolbar, could it simply be that they have 10,000 active users who use the vote buttons? Or would the numbers be far larger if this was the case.

reseller

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 11:56 am on Jul 11, 2007 (gmt 0)

scotland

It was mentioned earier about the Google Toolbar, could it simply be that they have 10,000 active users who use the vote buttons?

Possibly the human evaluators work is related to Random Query Evaluation [google.com].

Minuteman

5+ Year Member



 
Msg#: 3390143 posted 4:04 pm on Jul 11, 2007 (gmt 0)

The "revelation" that they have so many human evaluators is refreshing and expected to me. Google is, in a Matrix or Terminator movie sort of way, about machines, computers, algorithms, etc replacing humans. I've always said that search engines aren't about search anyway, they are about advertising. They perfected algorithms not to give me a "free" search engine but to keep them from embarrassingly putting a Victoria Secret ad on my Cub Scout website.

Google needs to maintain its aura of supreme ad context management to maintain its market position. To rely on human editors, after all, is an admission that Google is just another Yahoo or DMOZ with a different story line. They wouldn't want that now, would they?

europeforvisitors



 
Msg#: 3390143 posted 5:18 pm on Jul 11, 2007 (gmt 0)

To rely on human editors, after all, is an admission that Google is just another Yahoo or DMOZ with a different story line.

Fortunately, they'll never have to make such an admission, since they don't "rely on human editors."

reseller

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 8:32 pm on Jul 11, 2007 (gmt 0)

From what I read, it seems Google is moving from algorithms-driven-search engine to the direction of semi-human-edited-search engine. As such its going to rely on human editors or human evaluators. I.e those 10.000 evaluators are going to affect Google's serps directly!

Here is [mattcutts.com] what some Googlers said:

- Marissa Mayer : “Up to today we have relied on automation, but I believe the future will be a blend of both, combing the scale of automation and human intelligence.”

- Larry Page: "Larry says search is finding content… and that Wikipedia found a better way to organize information. he seems to like the model of using humans and process and machines."

- Matt Cutts: "So I think too many people get hung up on “Google having algorithms.” They miss the larger picture, which (to me) is to pursue approaches that are scalable and robust, even if that implies a human side. There’s nothing inherently wrong with using contributions from people–you just have to bear in mind the limitations of that data."

- Udi Manber said [stonetemple.com]: "If somebody says they don’t like a particular search result that’s a signal. So, we’ve been using that for a long time, and we are working on new ways of using it."

Its therefore of importance to know exactly which 10.000 human evaluators Marissa Mayer was referring to in her session with Dare Obasanjo. Was she referring to Google internal employees or the $10 - $15 temp evaluators worldwide?

[edited by: reseller at 8:56 pm (utc) on July 11, 2007]

europeforvisitors



 
Msg#: 3390143 posted 8:56 pm on Jul 11, 2007 (gmt 0)

From what I read, it seems Google is moving from algorithms-driven-search engine to the direction of semi-human-edited-search engine.

You're reading your own preconceptions into what was actually said. There's no way that a "semi-human-edited" search engine would be "scalable," to use a popular Google term. Where human input can be valuable is in providing cues and clues for writing and/or training the software that does the work. (And, of course, Google has been doing that all along; PageRank is based on human input, and it's highly unlikely that Larry Page and Sergey Brin let their original algorithm write itself back in the late 1990s.)

ADDENDUM: The proof is in the pudding, and having paid evaluators to help identify spam, distinguish between good and bad results, etc. should improve Google's search results over the long term--regardless of where they're being recruited, how much they're being paid, and whether their input is being used to train software, to facilitate research, or (less likely) to simply whack pages that don't deserve to be in Google's SERPs.

reseller

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 10:19 pm on Jul 11, 2007 (gmt 0)

EFV

..and having paid evaluators to help identify spam, distinguish between good and bad results, etc. should improve Google's search results over the long term--regardless of where they're being recruited, how much they're being paid,

Its not only how much they are being paid which worry me. Its a statement like this one [webmasterworld.com] too:

"they pay around $12 - $15 an hour, work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites."

Would any site owner and webmaster feel comfortable knowing that such kind of $12 - $15 evaluators might affect directly or indirectly which site to stay and which site to be deindexed on Google index?

europeforvisitors



 
Msg#: 3390143 posted 10:30 pm on Jul 11, 2007 (gmt 0)

Would any site owner and webmaster feel comfortable knowing that such kind of $12 - $15 evaluators might affect directly or indirectly which site to stay and which site to be deindexed on Google index?

I imagine that depends, to a great extent, on whether the site owner or Webmaster has anything to hide. :-)

ogletree

WebmasterWorld Senior Member ogletree us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3390143 posted 11:16 pm on Jul 11, 2007 (gmt 0)

The problem with this is a bunch of people will blame any problems they have to human evaluators. Just like they blame penalty's or any other theory on here. Most of the time it just comes down to quality links and content or obvious spam tactics.

sitetruth



 
Msg#: 3390143 posted 11:51 pm on Jul 11, 2007 (gmt 0)

That patent isn't about using human evaluators for rating. It's about developing a classification system for search queries ("query themes") and pages: "For the query theme "sites that provide free software downloads," the rule may be the requirement that the query contain the word "free" and "download." For the query theme "sites that help in finding an accommodation," the rule may be the requirement that the query contain one of the words: {"accommodation," "lodging," "hotels," . . . } and also that the query contains the name of a place (e.g., by matching one of a list of place names)." That's a scheme to detect special cases and route them off to a special search system. Yahoo was first to do that, with about 30 special cases (celebrities, sports, etc.), and Google is now moving in that direction.

There are patents for schemes which explicitly use user feedback to adjust ratings, and most of those patents are from Neelakantan Sundaresan at IBM Almaden Research, not from Google.

The fundamental trouble with using user feedback to drive search is that it's too easy to spam. Unless the number of raters per thing rated is high, as for movies, or you have to do an actual transaction with real money to submit a rating, as with eBay, most ratings will come from involved parties. Which does not help.

robertskelton

10+ Year Member



 
Msg#: 3390143 posted 12:37 am on Jul 12, 2007 (gmt 0)

I was one of the evaluators for a while. There's no way they have 10,000 - she must've been cleverly making it sound that way, whereas the 10,000 figure is for all Google employees.

They give multiple evaluators the same work, so that they get a consensus answer. Very basic "is this page spam or not" work. The purpose is to check examples of spam, not all spam. 10,000 evaluators would imply that they are making dozens of tweaks to the algorithm every day, and have 500 staff doing the tweaks... Google's search algorithm is too important to have that many people working on it at once.

I'd guess the real figures to be more like 20 working on the algorithm, and 100-300 evaluators.

europeforvisitors



 
Msg#: 3390143 posted 1:42 am on Jul 12, 2007 (gmt 0)

The problem with this is a bunch of people will blame any problems they have to human evaluators.

I don't think Google would regard that as a problem. They must be pretty thick-skinned by now. :-)

reseller

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 7:42 am on Jul 12, 2007 (gmt 0)

robertskelton

There's no way they have 10,000 - she must've been cleverly making it sound that way, whereas the 10,000 figure is for all Google employees.

Who knows what are those 10.000 are (:(

pexcornel

5+ Year Member



 
Msg#: 3390143 posted 12:21 pm on Jul 12, 2007 (gmt 0)

Let's just ask them what
AASearch
AAPub
AAShop
means ;)

reseller

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3390143 posted 12:38 pm on Jul 12, 2007 (gmt 0)

robertskelton

Btw, did Google do anything to secure the objectivity (fairness) of the evaluators work? any special requirements?

pexcornel

5+ Year Member



 
Msg#: 3390143 posted 12:42 pm on Jul 12, 2007 (gmt 0)

or at least secure the tool :))

Bolotomus

5+ Year Member



 
Msg#: 3390143 posted 1:37 pm on Jul 12, 2007 (gmt 0)

I've been saying this for years.

The bottom line is that 90% of the SEO tricks that we all learned in 1999 are worthless today. You cannot fool Google into thinking that your site deserves to be #1 in the results through some kind of trickery. Maybe you can temporarily, but in the end, you better have a site that looks good and works good and that people like. If you don't, all the meta-tag tweaking in the world isn't going to make a difference.

For years my company has been designing sites, and after a year or two online, the customer comes back to us saying "This is incredible! Google LOVES us!" Now we know why. Because the sites we design are good, they look good, they work good, and when the human evaluator comes to look at the site for even a minute or two, they give it the thumbs up.

It's been so obvious that they do this, just by watching Google's behavior. People said I was crazy, they said that there are too many websites out there to do this, they said that Google has publically stated that they seek to completely automate their process. But I don't care what anybody says--I care about what their software does. And their software has been screaming of human interference loud and clear ever since the day they went online.

That's why I feel completely justified in screaming "HAH! I TOLD YOU SO!" at the top of my lungs right now.

Cheers,
Bolotomus

Gimp

5+ Year Member



 
Msg#: 3390143 posted 2:58 pm on Jul 12, 2007 (gmt 0)

Could it be that when a reinclusion request is filed, several human evaluators give their opinions?

Hobbs

WebmasterWorld Senior Member hobbs us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3390143 posted 3:25 pm on Jul 12, 2007 (gmt 0)

Scan your logs for:

google.com/evaluation/search/rating/task-edit?task=

Get the IP, and find all hits from that IP in your logs:

Here is what I am seeing from the same (non Google) IP:

- Page1 is requested with a direct hit, no referrer (He is browsing the web)
- 43 min. later same page is hit giving the evaluation as referrer (he decided to evaluate my page)
- 30 seconds later the same page is hit, no referrer (He hit an external link on the page and then the back button)
- 2.5 Hours later same page is hit giving the evaluation as referrer (I have no idea why)
- He bookmarks the page (Plans to visit again)

And this is a very minor trivial page on my site, the same IP does not surf to other links on the page, does not try to find out more about my site or what it does, just evaluating the page independently from anything else, I have also looked up everyone else that requested the page since and it is not visited by any Google IP or another evaluation referrer.

[edited by: Hobbs at 3:33 pm (utc) on July 12, 2007]

pexcornel

5+ Year Member



 
Msg#: 3390143 posted 5:01 pm on Jul 12, 2007 (gmt 0)

Oh Hobbs You wouldnt belive what I found in my logs from Google! :)
I'll just say this:
"We really appreciate your contribution to Ad Quality Check :-) "
:D

This 106 message thread spans 4 pages: < < 106 ( 1 2 [3] 4 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved