Forum Moderators: Robert Charlton & goodroi
Q: How do they tell if they have bad results?
A: ...they have 10,000 human evaluators who are always manually checking the relevance of various results.Article by attendee Dare Obasanjo [25hoursaday.com]
Next time we're trying to figure out some odd change in the SERPs, we might do well do remember this human factor, eh?
When you wrote this:
We have. She is referring to the fact that she considers every employee at Google a "search evaluator". How to use Google, and how to report poor quality search results is part of orientation for any new employee. If you are a googler and find poor quality search results, it is your duty to report it.
are you referring to the following section of Dare Obasanjo session with Marissa Mayer?
Q: How do they tell if they have bad results?
A: They have a bunch of watchdog services that track uptime for various servers to make sure a bad one isn't causing problems. In addition, they have 10,000 human evaluators who are always manually checking teh relevance of various results.
Looking at click patters on their site after SERPS change and comparing the number of people click to page 2, or the number that refine their search would gauge result quality.
Looking at toolbar or analytics data for time on a site after a search would also be insightful.
The best data that Google could possible have is the feedback provided by end users in the form of clicks, time on a given result site, and whether they return to try a different result. Google has for a number of years released products that give them access to just this type of data (toolbar and analytics being notable examples). I think the judgment of the masses would (in this case) provide much better data then the opinion of a handful of quality evaluators.
this kind of release could be seen as a propaganda type response to questions like mine.
i don't believe it at all. i seem to remember matt saying that the webspam team had 200-300 in it. he said that at the time vanessa fox left. all this is based on recollection..
Search Quality Coordinator - HyderabadPosition based in Hyderabad, India.
Do you have a passion for Google? Do you desire to help improve the quality of Google's search results? Google is recruiting enthusiastic, web-savvy individuals for search quality evaluation.
Responsibilities:
* Reviewing assigned sites for quality and content.
* Troubleshooting website issues and identifying areas of concern and interest.
* Investigating web sites.
* Working on special projects, as needed.Requirements:
* Excellent web research skills.
* Excellent analytical skills.
* Detail oriented; ability to complete large volume of work quickly.
* Flexible.
* Proven track record of exceptional performance, high productivity and meeting deadlines.
* Ability to work cooperatively and proactively with team members.
* Fluency in English.A plus:
* One to three years related experience in an Internet company and with web research.
* Previous experience with a computer programming language.
* Familiarity with typical web practices such as managing a domain name.
* Basic HTML experience.
10,000 may seem like a lot but it's not enough
obviously not! or, and that is a guess, the feedback can not be integrated as it should be.
- "artificial sub domain link boost" works (at least for some major brands I watch)
- the "domain age advantage" is much too strong
- "brainless link bait" produces too good results for a human filter system, IMHO
and some other quality glitches I see, when I google around vote against a 10k human ranking factor.
also, the quote says "... user feedback... is a signal ...", which I understand as another of many signals finally needed to be digested by the algo.
I can not believe, that hiring 100 helping hands myself (which a lot of professionals here could afford) and making them file complaints about specific sites would make any difference in the rank of that site, if all other signals are OK!
this smells like another "we need good press about our quality" approach by a company that re-invented online marketing.
IMHO, the truth lies somewhere between all the facts and fictions around the algo and comments from single googlers.
So, a signal? Yes! More than 10k human rankers? Probably part time, summing up to 500 full-time rankers... maybe even less. I think I could clean out 5 domains per hour (when i work really slow) without real hard work, just using a solid commercial keyword list.
500 full-time human editors could do 2500 domains per hour, which would sum up to 80,000 spam domains wiped from the index per week, yet there are such poor results to find? Either the signals are very very weak or the editors are really really bad!
2 cents,
P!
I'd hate to compete in the serps with one of those with all the inside information and connections they have, sort of like dmoz all over again.
Even if they do have 10,000 quality evaluators, a lot of those people may be (and probably should be) part-timers.
[edited by: wildbest at 5:46 pm (utc) on July 10, 2007]
10,000 human evaluators
Do these 10,000 evaluators have the power to make manual serp changes? If so, that is scary.
sort of like dmoz all over again
My thoughts exactly.
Do these 10,000 evaluators have the power to make manual serp changes?
Not according to the patent [webmasterworld.com]. You're right, that would be a scary scenario.
Any guesses as to the salary?
According to this article [searchbistro.com]:
The Google testers are paid $10 - $20 for each hour they filter the results of Google. Payment is done through Payroll. The international agents are instructed by the Human Quality Evaluation Team of Google.
I did have visits from them on a new site something like 2-3-4 months ago. I spotted them as I was monitoring the logs of the new site. I recall seeing something like 5 visits from various computers based in my country, and the referrer contained something like G evaluation.
My country (former Soviet bloc) is not a big one, population is about 1/30 of that of the US. If my local language site could receive 5 evaluators (probably not everyone in G's local team visited my site), the number of staff in larger markets might be magnitudes larger.
To answer the question about being able to change SERPS, that was not an option.. =)
We were told our data would be used for research purposes and would not directly affect the search results.
It is my opinion that our rating data was/is used to help better tweak their algorithm.
Did you get the job at that time through a similar job announcement like this one?
QUALITY RATER - (SPANISH, DUTCH, ITALIAN, FRENCH) This is a temporary role offered through Kelly Services. Google Inc. is recruiting part-time, temporary, home-based workers to help with work on a search quality evaluation on a project basis. You would work at your own pace, and the time and length of any particular work session would be up to you. Candidates will evaluate search results and rate their relevance....
< I shortened the quote
see Terms of Service [webmasterworld.com] >
[edited by: tedster at 7:49 pm (utc) on July 10, 2007]
