homepage Welcome to WebmasterWorld Guest from 54.161.214.221
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
More Details on Google's Human Evaluators
tedster




msg:3745341
 11:32 pm on Sep 15, 2008 (gmt 0)

A little tidbit for those of us who are curious about Google's human evaluators (employed in many countries and languages, by the way.)

These evaluators are carefully trained and are asked to evaluate the quality of search results in several different ways.

1) We sometimes show evaluators whole result sets by themselves
2) or "side by side" with alternatives
3) in other cases, we show evaluators a single result at a time for a query and ask them to rate its quality along various dimensions.

Offical Google Blog [googleblog.blogspot.com] [numbering added by tedster]

The only human evaluation I was certain of before was the first one. The idea was that a group of independent editors would each evaluate a (relatively competitive) SERP and give their suggestions for whether a URL in the result was too high, too low, or just right. Only if there was agreement across these independent evaluators would a plus or minus factor get integrated into the ranking for that URL. Differences of opinion would get kicked up to a supervisor level.

These other two approaches would need a different statistical method if the end result is a tweak of the algorithmic rankings. They don't seem to be directly mentioned in the Human Editorial Input Patent [webmasterworld.com].

 

tedster




msg:3745476
 6:53 am on Sep 16, 2008 (gmt 0)

I've read through a couple times now, and it seems to me that the human evaluator team may be used more in the evaluation of low volume experimental SERPs than I was originally thinking.

Seb7




msg:3746163
 8:56 am on Sep 17, 2008 (gmt 0)

I may be wrong, but didnít Google say their SERPs have no human intervention? Indirectly, there is lots of human intervention going on. And what happens if one of these evaluators have a little side website business? How much influence do they have?

rise2it




msg:3746170
 9:12 am on Sep 17, 2008 (gmt 0)

If all they were doing was 'quality' - meaning kicking out the 50,000 page fluff sites, that would be fine.

Otherwise, you're getting into human bias issues - and even those with PHD's have them.

So, now you have someone (subconsciously) choosing one site over another based on color or layout or logo, instead of content and ease of use.

maximillianos




msg:3746248
 12:22 pm on Sep 17, 2008 (gmt 0)

It was my understanding that the QA work these folks do would not directly influence SERPs, however it would be used as research to improve their algorithm.

Or at least this is what we were told...

Also, you have to disclose all your web properties before you can be accepted into the position, and they do their homework to verify.

Beyond that, you never act alone in the process of rating the quality of the results. Your "vote" is always compared randomly to many other votes on the same content to make sure you are not being bias...

The general thinking is that if 10 people say something is good, and one person says it is bad... that person's vote is really not counted... They had some interesting moderation techniques in place that would resolve such problems.

Personally, as a webmaster myself, I was very impressed with the process, and felt it was definitely fair and had adequate checks and balances.

marketingmagic




msg:3746253
 12:41 pm on Sep 17, 2008 (gmt 0)

and how do I contact this team? :-)

pageoneresults




msg:3746254
 12:57 pm on Sep 17, 2008 (gmt 0)

For those of you following this topic, you may want to read the original Google "Guidelines for Quality Raters" document that is now available on the Internet. It is an interesting read at 44 pages long. I printed it the day it was made public last year I believe and read it when time permits. It is worth the read! ;)

Miamacs




msg:3746264
 1:17 pm on Sep 17, 2008 (gmt 0)

n years after the deployment of the current evaluation system they're finally being open about it - in a way.

btw thanks for the blog post Scott.

But note that this is not the 'site' evaluation every other webmaster here fears so much, it's SERP evaluation... basically QA

It's about the process of:
'Algo tweak/layout testing, Step 1.5: closed beta'

...

Of course with the system in place it could be used for many things.
Mind you, if a site was cr@p enough to get a manual penalty ( accidents, hacks, user spam aside )...

...i hardly have any sympathy for it and am glad to see it out of the index.

...

hannamyluv




msg:3746273
 1:29 pm on Sep 17, 2008 (gmt 0)

I personally know a Google human evaluator. She is given a query and a page. She is asked to rate how closely the page matched the given query. I would assume that the page was one that came up in the results, but as it was presented to her, this was not the case. She got a phrase and a page and was asked to judge how well the page matched the phrase.

sgietz




msg:3746328
 2:27 pm on Sep 17, 2008 (gmt 0)

So is this like tasting ice cream for a living? Where do I sign up?

maximillianos




msg:3746377
 3:10 pm on Sep 17, 2008 (gmt 0)

So is this like tasting ice cream for a living? Where do I sign up?

It is not all that easy... =) I did it for about 3 months a few years back. It was fun for about the first week, then it got really boring and tedious.

They try and offer you a number of projects you can work on, so you can mix your time up... But for folks like most of us (webmasters) you might find the work very boring after a while.

For the pay, it was not worth it to me to keep the job. Of course at the time I was working a full-time job (corporate america), running my websites AND doing the Quality Rater job... that could have been why I had no time for it.

Perhaps now that I no longer work in corporate america it would serve as a nice mind-numbing escape for a few hours a day... ;-)

Gomvents




msg:3746525
 5:36 pm on Sep 17, 2008 (gmt 0)

maximillianos, how much did it pay back when you did it? I think they are offering much less now...

maximillianos




msg:3746861
 4:43 am on Sep 18, 2008 (gmt 0)

maximillianos, how much did it pay back when you did it? I think they are offering much less now...

I am not sure if my memory is correct, but I want to say it was like $15/hr.

Robert Charlton




msg:3746884
 6:32 am on Sep 18, 2008 (gmt 0)

It is an interesting read at 44 pages long. I printed it the day it was made public last year I believe and read it when time permits. It is worth the read!

Most definitely worth the read. It's a very well thought out system, I feel, and has led me to be much more demanding of writers who create content for the sites I work on.

restless




msg:3747033
 12:40 pm on Sep 18, 2008 (gmt 0)

So what happens when a site is flagged as non relevant and gets demoted in it's ranking, so then the webmaster decides to make their page more relevant by adding more content, does google pick up the change and notify this human team to re-evalute?

internetheaven




msg:3747373
 6:52 pm on Sep 18, 2008 (gmt 0)

and how do I contact this team? :-)

These guys must have more protection than a juror in a mafia trial ...

fischermx




msg:3748147
 6:51 pm on Sep 19, 2008 (gmt 0)

They might be obligated to first say they work as strippers rather than say their real job.....hehe

ReSiever




msg:3749510
 2:06 pm on Sep 22, 2008 (gmt 0)


So what happens when a site is flagged as non relevant and gets demoted in it's ranking, so then the webmaster decides to make their page more relevant by adding more content, does google pick up the change and notify this human team to re-evalute?

Important question. Once you have been flagged, is there any way to re-improve your rankings again? Will the system automatically pick up the changes, will the page be evaluated again after changes have been made and noticed, or do people need to fill in a reinclusion report?

Anyway, I believe this these flags are directly related to the -5, -10, -30, -950 penalties. When your page is relevant, you won't notice anything, unless another page is more relevant or maybe even vital. But when it comes to the flags 'useful', 'relevant', 'not relevant' and 'off topic', i strongly believe they are directly related to these 'mysterious' penalties. Off topic? Off you go down the drain to -950. Not relevant? Then no need to be in the top 30 for that word, so -30.

Thats my opinion....

tedster




msg:3749565
 2:58 pm on Sep 22, 2008 (gmt 0)

My experience is that the -950 is more of a "too perfectly relevant" problem - an over-optimization penalty. You may well be right about some of the others, however.

ReSiever




msg:3749589
 3:15 pm on Sep 22, 2008 (gmt 0)

My experience is that the -950 is more of a "too perfectly relevant" problem - an over-optimization penalty. You may well be right about some of the others, however.

True, over-optimization also can be a factor that comes into play when you're dealing with the -950 penalty. Maybe i should have used the minus 5, 10, 30 and 60 instead in this case ;-)

Nevertheless, i've seen these kinds of 'penalties', and in many cases a human evaluator visited the pages short before the rankings dropped down a page or 2, 3 in the SERPS.

tedster




msg:3749616
 3:49 pm on Sep 22, 2008 (gmt 0)

It strikes me that there are two different types of human evaluation. One, which this thread focuses on, involves a TEAM of evaluators looking at a set of results for a given query. It takes consensus across the independent members of this team to move a ranking up or down - and up is always possible. I'm also not sure that members of the Evaluator Army would be visiting from a Google IP address. Maybe, but this is a globally located bunch of folks.

There can also be a manual visit from an engineer on Google's search quality team - rather than the human editorial army. They would be checking on something that was flagged for inspection by any number of means - algorithmic, competitor reports, and so on.

This search quality engineer has a lot more clout and might be able to instigate some action as an individual, subject to a supervisor's approval of course.

What I weant to emphasize is that the "patented" human editorial input approach is not the only cause of a manual inspection visit showing up in your server logs.

ReSiever




msg:3749707
 5:56 pm on Sep 22, 2008 (gmt 0)

I agree, the patent partly describes what is going on with their 'investigators'.

You are right about the two types of evaluation, tedster. They do have their teams across the world, and the individual Google engineers. From what i have been seeing in our logfiles, the evaluators not only come from Google IP's, but I also see other, non-Google-IP adresses from which the evaluators do their thing. Their useragents are diverse, and, for what i can see, only a few of them have the right to 'pull the trigger' and finally flag your webpage, or entire site. And when they flag it to be spam, i certainly believe that Google Engineers will then come by and pay your site a visit to see what they can do to improve their algorithm.

anallawalla




msg:3750173
 5:41 am on Sep 23, 2008 (gmt 0)

If you search for "Google jobs evaluator" there are a few vacancies all over the globe (I didn't check to see if they are old entries, so check the cached copy).

Bestpublisher




msg:3753851
 6:40 pm on Sep 28, 2008 (gmt 0)

Thank you so much for the information.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved