Forum Moderators: Robert Charlton & goodroi
Q: How do they tell if they have bad results?
A: ...they have 10,000 human evaluators who are always manually checking the relevance of various results.Article by attendee Dare Obasanjo [25hoursaday.com]
Next time we're trying to figure out some odd change in the SERPs, we might do well do remember this human factor, eh?
lets be honest anything apart from search has been a huge, huge, huge (revenue) failure for Google.
That sounds like a topic for the GOOG forum, not the Google Search News forum, but in any case, the statement is completely wrong, as one can see by reading any Google earnings report.
It's just a matter of improving organisation.
So we need to believe that G will get progressively more serious about content and linking technique control. When this is complete few sites will remain in their positions , by links alone and content alone. What will evolve are techniques relying solely on background data comprising such comparative things as adherence to guidelines and user activity measurements of quality.
These types of things are the new voting paramaters for a website. I think links will become a thing of the past, and I recall some recent posts indicating these thoughts to.
This does mean that G will move to a new level of complete dominance IMO
These types of things are the new voting paramaters for a website. I think links will become a thing of the past, and I recall some recent posts indicating these thoughts to.
I don't think links will become a thing of the past with Google; they just won't be as powerful by themselves as they used to be, and Google will continue to improve its ability to distinguish between valuable and meaningless links.
I think those human evaluators have a say on the ranking of the top few hundred sites..
You could pretty well kill the bias rate by recycling the same SERP to a variety of reviewers, wisdom of crowd type of thing.
they pay around $12 - $15 an hour, work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites.
Thanks for feedback.
they pay around $12 - $15 an hour, work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites.
And that might explain the poor quality of Google's serps that have been mentioned on different threads on WebmasterWorld for the latest years.
We know that multi billion Google could afford to spend more to improve its serps quality but they haven't unfortunately.
IMO, better algos driven serps than the $12 - $15 evaluators you and another friends have mentioned on this thread.
Talking about the human-role-in-Google :-)
I'm sure that $12 - $15 is top money also in Mogadishu in Somalia for "work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites."
Oh well...you get what you pay for :-)
For me the role of human in Google is better explained in the great efforts of kind Googlers like Matt Cutts, Adam Lasnik and Vanessa Fox to communicate with the webmasters communities on forums and during conferences. Power to Matt, Adam and Vanessa!
It was mentioned earier about the Google Toolbar, could it simply be that they have 10,000 active users who use the vote buttons?
Possibly the human evaluators work is related to Random Query Evaluation [google.com].
Google needs to maintain its aura of supreme ad context management to maintain its market position. To rely on human editors, after all, is an admission that Google is just another Yahoo or DMOZ with a different story line. They wouldn't want that now, would they?
To rely on human editors, after all, is an admission that Google is just another Yahoo or DMOZ with a different story line.
Fortunately, they'll never have to make such an admission, since they don't "rely on human editors."
Here is [mattcutts.com] what some Googlers said:
- Marissa Mayer : “Up to today we have relied on automation, but I believe the future will be a blend of both, combing the scale of automation and human intelligence.”
- Larry Page: "Larry says search is finding content… and that Wikipedia found a better way to organize information. he seems to like the model of using humans and process and machines."
- Matt Cutts: "So I think too many people get hung up on “Google having algorithms.” They miss the larger picture, which (to me) is to pursue approaches that are scalable and robust, even if that implies a human side. There’s nothing inherently wrong with using contributions from people–you just have to bear in mind the limitations of that data."
- Udi Manber said [stonetemple.com]: "If somebody says they don’t like a particular search result that’s a signal. So, we’ve been using that for a long time, and we are working on new ways of using it."
Its therefore of importance to know exactly which 10.000 human evaluators Marissa Mayer was referring to in her session with Dare Obasanjo. Was she referring to Google internal employees or the $10 - $15 temp evaluators worldwide?
[edited by: reseller at 8:56 pm (utc) on July 11, 2007]
From what I read, it seems Google is moving from algorithms-driven-search engine to the direction of semi-human-edited-search engine.
You're reading your own preconceptions into what was actually said. There's no way that a "semi-human-edited" search engine would be "scalable," to use a popular Google term. Where human input can be valuable is in providing cues and clues for writing and/or training the software that does the work. (And, of course, Google has been doing that all along; PageRank is based on human input, and it's highly unlikely that Larry Page and Sergey Brin let their original algorithm write itself back in the late 1990s.)
ADDENDUM: The proof is in the pudding, and having paid evaluators to help identify spam, distinguish between good and bad results, etc. should improve Google's search results over the long term--regardless of where they're being recruited, how much they're being paid, and whether their input is being used to train software, to facilitate research, or (less likely) to simply whack pages that don't deserve to be in Google's SERPs.
..and having paid evaluators to help identify spam, distinguish between good and bad results, etc. should improve Google's search results over the long term--regardless of where they're being recruited, how much they're being paid,
Its not only how much they are being paid which worry me. Its a statement like this one [webmasterworld.com] too:
"they pay around $12 - $15 an hour, work from home, part time, no benefits. the people are not highly specialized, just trained to look for obvious problems, and rate sites."
Would any site owner and webmaster feel comfortable knowing that such kind of $12 - $15 evaluators might affect directly or indirectly which site to stay and which site to be deindexed on Google index?
Would any site owner and webmaster feel comfortable knowing that such kind of $12 - $15 evaluators might affect directly or indirectly which site to stay and which site to be deindexed on Google index?
I imagine that depends, to a great extent, on whether the site owner or Webmaster has anything to hide. :-)
There are patents for schemes which explicitly use user feedback to adjust ratings, and most of those patents are from Neelakantan Sundaresan at IBM Almaden Research, not from Google.
The fundamental trouble with using user feedback to drive search is that it's too easy to spam. Unless the number of raters per thing rated is high, as for movies, or you have to do an actual transaction with real money to submit a rating, as with eBay, most ratings will come from involved parties. Which does not help.
They give multiple evaluators the same work, so that they get a consensus answer. Very basic "is this page spam or not" work. The purpose is to check examples of spam, not all spam. 10,000 evaluators would imply that they are making dozens of tweaks to the algorithm every day, and have 500 staff doing the tweaks... Google's search algorithm is too important to have that many people working on it at once.
I'd guess the real figures to be more like 20 working on the algorithm, and 100-300 evaluators.
The problem with this is a bunch of people will blame any problems they have to human evaluators.
I don't think Google would regard that as a problem. They must be pretty thick-skinned by now. :-)
There's no way they have 10,000 - she must've been cleverly making it sound that way, whereas the 10,000 figure is for all Google employees.
Who knows what are those 10.000 are (:(
Btw, did Google do anything to secure the objectivity (fairness) of the evaluators work? any special requirements?
The bottom line is that 90% of the SEO tricks that we all learned in 1999 are worthless today. You cannot fool Google into thinking that your site deserves to be #1 in the results through some kind of trickery. Maybe you can temporarily, but in the end, you better have a site that looks good and works good and that people like. If you don't, all the meta-tag tweaking in the world isn't going to make a difference.
For years my company has been designing sites, and after a year or two online, the customer comes back to us saying "This is incredible! Google LOVES us!" Now we know why. Because the sites we design are good, they look good, they work good, and when the human evaluator comes to look at the site for even a minute or two, they give it the thumbs up.
It's been so obvious that they do this, just by watching Google's behavior. People said I was crazy, they said that there are too many websites out there to do this, they said that Google has publically stated that they seek to completely automate their process. But I don't care what anybody says--I care about what their software does. And their software has been screaming of human interference loud and clear ever since the day they went online.
That's why I feel completely justified in screaming "HAH! I TOLD YOU SO!" at the top of my lungs right now.
Cheers,
Bolotomus
google.com/evaluation/search/rating/task-edit?task=
Get the IP, and find all hits from that IP in your logs:
Here is what I am seeing from the same (non Google) IP:
- Page1 is requested with a direct hit, no referrer (He is browsing the web)
- 43 min. later same page is hit giving the evaluation as referrer (he decided to evaluate my page)
- 30 seconds later the same page is hit, no referrer (He hit an external link on the page and then the back button)
- 2.5 Hours later same page is hit giving the evaluation as referrer (I have no idea why)
- He bookmarks the page (Plans to visit again)
And this is a very minor trivial page on my site, the same IP does not surf to other links on the page, does not try to find out more about my site or what it does, just evaluating the page independently from anything else, I have also looked up everyone else that requested the page since and it is not visited by any Google IP or another evaluation referrer.
[edited by: Hobbs at 3:33 pm (utc) on July 12, 2007]
