An Inside Look at Algorithm Changes: Amit Singhal keynote at SMX London

[netmeg]

Near the end of the talk, someone asked if how much money Google will make is factored into decisions about changes to Google’s (unpaid search algorithms). Singhal was adamant: “no revenue measurement is included in our evaluation of a rankings change.”

[searchengineland.com...]

Interesting stuff.

[goodroi]

<sarcasm>This aint important, its just an inside look at what this guy is thinking. It doesn't matter that this guy was behind some of the biggest Google updates and is also working on even bigger future updates. Why would I want to know what this guy has to say?</sarcasm>

Seriously, this is a good article to read for anyone that wants to better understand Google.

[netmeg]

Well that's what I thought, but it kind of got buried behind all the other usual stuff.

[Leosghost]

If Vanessa was reporting Amit's exact words words here, or quoting an old Google "explanation", or using her own words..

2.That algorithm change is run on a test set of data and if all looks good, human raters look at before and after results for a wide set of queries (a kind of manual A/B test). The human raters don’t know which is the before and which is the after. The raters report what percentage of queries got better (more relevant) and what percentage got worse (less relevant).

Then either she or Amit or Google's PR dept don't understand logic ( could be any one, or all of those three )..or someone ( again, could be any one or all of those three ) is being disingenuous* and hoping that the "algo babble" will dull our critical faculties and we won't notice the impossibility of item #2..

If one does not know the starting point ( one does not know which is the "before" and which is the "after" ) one cannot possibly give a value judgment as to whether the result got better or worse..better or worse are relative terms, which require that one knows "in relation to what they previously were"..in other words, to a starting point, or simply , to what they were before..

So either item #2 is untrue, or is nonsense..

Unless one knows what the result was for a given query ( the "before" ) one ( human rater ) cannot decide if the new result was better ..or worse..

If Google and or their ex employees cannot grasp that simple concept..it may well explain why sometimes the pages presented in the top positions in certain serps are totally irrelevant to the query, or lead to pure spam or 404s , and why there does appear from time to time to be a large element of "throw it at the serps and see what clicks" under the guise of "quality" and or "relevance" "improvements" with certain major updates and their effects in certain verticals..

Other than that "twilight zone logic squib", Amit didn't say anything that he or another Google spokesperson has not said before, re tweaks, algo changes etc..

*Could also be that in the haste to get some press coverage in the week of the facebook IPO..nobody thought to check the "Don't all look at that.. Look over here" ..text for "must not contain self contradictory statements".."never attribute to mendacity what can be" ..etc ...

[AndyA]

Quote from article referenced in netmeg's original post:
Near the end of the talk, someone asked if how much money Google will make is factored into decisions about changes to Google’s (unpaid search algorithms). Singhal was adamant: “no revenue measurement is included in our evaluation of a rankings change.”

If that's true now, it won't be for long. Since Google is a public company, it has an obligation to its stockholders to show a profit. If income drops due to changes in Google SERPs, you can be certain the stockholders will demand Google do something about it.

If Facebook and Bing continue to expand their relationship, I can see the possibility of that being a game changer for Google. So many people are logged in to Facebook for much of the day, if their searches default to Bing while logged in, I suspect many won't change that because it will be easier. For all its problems, lots of people love Facebook, and with Bing's new SERPs showing what their friends recommend, that has a huge impact on relevancy to the searcher.

[rlange]

Leosghost wrote:
If one does not know the starting point ( one does not know which is the "before" and which is the "after" ) one cannot possibly give a value judgment as to whether the result got better or worse..better or worse are relative terms, which require that one knows "in relation to what they previously were"..in other words, to a starting point, or simply , to what they were before..

So either item #2 is untrue, or is nonsense..

Neither. Item #2 just doesn't look like it was worded as well as it could have been.

The first set they are given is the starting point. When they say "got worse" or "got better", that's a statement of the quality of the second set compared to the first set.

It's just the usual, blind "taste" test.

--
Ryan

[Leosghost]

If they don't know which was the first set..then they cannot decide which was "better"..better is a linear concept..they can only decide which they prefer..
and as the pages on the web are constantly changing and interacting and linking to reach other ..they have at best been given a "snapshot"..and a second "snapshot" and asked which they prefer..

Meanwhile the snapshots have become stale and irrelevant as the interactions of the items contained within the "snapshots" with the rest of the web have modified them and their relative relevancy to any query..

They are trying to paint zebra stripes on enormous herds of galloping equines all crossing each others paths and modifying their own trajectories as a result of that interaction.. and decide which design they prefer..which is why they so frequently make a mess..

And I say that as someone who is usually quite happy with the results I get to queries within Google ( but then none of us here search the same way as the average user does )..and also as someone whose websites are highly placed, so who is not complaining about their methodology due to any feelings of being penalised..nor am I suffering from any paranoia or Google hate..

Their method for deciding "did serps get better as a result of us doing X or Y or Z or any combinations of tweaks" is flawed..if it is as described by the article..

Possibly better methods come to mind..

Google knows where and how to get in touch with me if they would like some alternatives..

There is a limit to how much creative thinking one gives away in public forum posts..

Especially if one is used to being paid for such..

@rlange..I was replying to the contents of your post prior to your editing of it ..your edited post brings our viewpoints on the article closer together,( while not in complete agreement ) I think..

[Shaddows]

@Leosghost
Unusually, we disagree on a purely analytical point- rlange is correct, the statement is badly worded. Try this:

That algorithm change is run on a test set of data and if all looks good, human raters look at before and after results for a wide set of queries (a kind of manual A/B test). The human raters don’t know which is the before and which is the after. The raters report which set was better, and hand it back. The markers then declare whether the lab rats prefer coke or new coke.

Edit- To paraphrase: I was replying to the contents of your post prior to your editing of it ..your edited post brings our viewpoints closer together and perhaps into agreement

Futher edit - Quote box

[Leosghost]

Indeed..:) I think re the wording ( and what should have been the wording of item 2 ) we can be said to be in broad agreement..( the statement is very badly worded ) :)

Even then.. if we take your improved wording of it as being a more accurate description of what they do..

That algorithm change is run on a test set of data and if all looks good, human raters look at before and after results for a wide set of queries (a kind of manual A/B test). The human raters don’t know which is the before and which is the after. The raters report which set was better, and hand it back. The markers then declare whether the lab rats prefer coke or new coke

...there are definitely better ways that they could deal with "did serps get better as a result of us doing X or Y or Z or any combinations of tweaks" ..

[brinked]

At the end of the day, he said, site owners need to take a hard look at what value their sites are providing. What is the additional value the visitor gets from that site beyond just a skeleton answer? Ultimately, it’s those sites that provide that something extra that Google wants to showcase on the first page of search results.

Something I have been saying ever since panda was released.

[superclown2]

At the end of the day, he said, site owners need to take a hard look at what value their sites are providing. What is the additional value the visitor gets from that site beyond just a skeleton answer? Ultimately, it’s those sites that provide that something extra that Google wants to showcase on the first page of search results.

It's a nice thought but it doesn't happen in practice. The verticals I'm interested in are still dominated by large brands with minimal content which is identical in meaning to the content on hundreds of other sites. There is no extra value there I'm afraid. Right now the best sites that I'm interested in (belonging to competitors as well as to myself) with lots of good, useful content are buried by the penguin whilst dross (including a lot of my experimental sites, I'm happy to say) has risen to take their places (but still below the brands, of course!)

If Google wishes to see good, useful websites produced then it would help if they recognised them, and rewarded them instead of taking the easy way out by rewarding brand, and thereby devaluing their own results. Will this make Google more money? I dunno but I reckon it will certainly lose them market share if it continues for much longer.

[g1smd]

The person presented with the before and after doesn't know which is which. They have to choose which is the better result.

Once they mark that, the people running the test can see if they chose the before or the after result.

Quote of the day was "I'm not going to the whiteboard and writing out the entire Google algorithm anytime soon". :)

[Leosghost]

Arggh g1smd :)..you can't choose the "better" without a "base" to be "better" than..

You can choose the one you "prefer" from a choice of two though..
One amongst many flaws in Google's approach is that Google have no way to be sure ( other than asking the "lab rats" ) if they have ever run a search for the terms whose results they are given to assess and assign a preference to..

and one of very many problems with that approach is ...

Most people's memory is appalling, and most make their decisions and base their preferences on highly subconscious factors which in the ad business we learned to manipulate in ways which they would be in some cases highly embarrassed to be aware of..

There is no way to attribute the one perfect answer to each of us unless we are the only one to assess the resulting answer..and our "preference" will change with the events that lead up to and surround that moment of "choice"..as will that of their "lab rat(s)"

Anything else is merely ignoring environment factors and history and personal experience , which get you the LCD acceptable response..

The LCD response discards or demotes any "outside the average" response..

Somewhat similar to "family fortunes"..where the winning answer /response is not necessarily the factually correct answer, but merely the one that the majority preferred or replied with as their answer..

"Our survey said".."creationism / intelligent design is a valid subject to be taught in schools"..neglecting the effect of day time TV, education level, geography,location, age, and the million and one environmental issues that condition the response..or what the person may have been listening to on the radio on the way to being surveyed..

In Zen the glass of water has a taste and an importance which depends entirely on the events leading up to ones encounter with it ..the environment in which one encounters it..and where one has already projected to be afterwards..

I can ( and do ) show and sell the same identical relatively expensive items to at least 5 distinct sections of society ( using different targeted websites )..at different prices..and using different names and descriptions..they ( each customer group ) have their own idea and preconception as to what the items are, what their significance is ..and what kind of people use or buy them..

They identify very strongly with their own group..and in some cases are very antagonistic to members of the other group(s)..and would not wish to be placed in a group other than the one they have chosen for themselves..

Quality and relevance is totally subjective in many areas..and Google's LCD crowd sourcing of "better" is harming their serps in many verticals, because the method is not applicable...for as I have said many reasons..

On another note..I doubt if Amit or anyone else could now write out the algo on a white board..their writing would be obsolete and as they began and it's inaccuracy would increase with each passing minute..

Their tweaks nowadays amount to herding cats ( of their own creation ) each cat armed with many paint brushes amongst a stampede of equines which the cats are attempting to paint as zebras and asking passers by "which, in your opinion is the cutest zebra" many of those asked will confuse paint covered cats with zebras..and many will be looking at the mongooses..or the "whatever"..but will reply nevertheless..

Survey groups also consist entirely of those willing to be surveyed..and who by definition wish to have their ideas take precedence..

Somewhat like the adage about politicians..that "anyone expressing an interest in running for political office should be barred from doing so"..:)

[LostOne]

Skeleton answer?

"Ultimately, it’s those sites that provide that something extra that Google wants to showcase on the first page of search results."

I've never bashed Google until now. That's the most inane comment ever from that guy. Jiminy crickets dude. Pull 'yer head...you know from where. Say I want information about a certain product. All I get from Google is sites with pictures and price tags.

Something extra? Extra what? What is so extra about those big brands that provide nothing unique. Now I'm really LOST

[Robert Charlton]

you can't choose the "better" without a "base" to be "better" than...

Agreed. The phrasing in the original article was misleading... and it's hard to say whether that was Vanessa paraphrasing Amit, or Amit paraphrasing his own thoughts during the live presentation (such things do happen)... or whether this is much more complex than meets the eye and/or that Amit's answer was purposely vague.

Here's the statement as published, with the debated sentence bolded...

2. That algorithm change is run on a test set of data and if all looks good, human raters look at before and after results for a wide set of queries (a kind of manual A/B test). The human raters don’t know which is the before and which is the after. The raters report what percentage of queries got better (more relevant) and what percentage got worse (less relevant).

The further I go into several of the fairly reasonable interpretations of the above, including several of my own, I'm increasingly skeptical of any one of them. ;)

I think we agree that the raters aren't specifying chicken or egg here... and I'm tending towards lrange/Shaddows interpretation... except that I'm wondering whether Google would rely only on subjective rater reactions, of broadly comparing result set-A to result set-B... even at this very early stage of tuning its "seed set" to adjust the prototype of an algorithm prior to "crowd sourcing".

Both the article quote and LG's arguments... along with the increased complexity of the algorithm... are suggesting something additional, and perhaps more granular than that, would be prudent. Here's a Google comment on human evaluators from 2008, before the more complex set of evaluator guidelines of 2011....

Search evaluation at Google
September 15, 2008
[googleblog.blogspot.com...]

Human evaluators. Google makes use of evaluators in many countries and languages. These evaluators are carefully trained and are asked to evaluate the quality of search results in several different ways. We sometimes show evaluators whole result sets by themselves or "side by side" with alternatives; in other cases, we show evaluators a single result at a time for a query and ask them to rate its quality along various dimensions.

In the early evaluation process, I'm thinking, there could well be some evaluation of individual results within the test sets according to the 2011 (or later) raters' guidelines, and these evaluations might then be compiled by Google.

A point from Vanessa's article does suggest that some care is taken during this early stage in the process...

3. This process gets looped several times as the algorithm is tweaked to better serve results for the queries in the "worse" set.

Note that steps 3, 4, and 5 described in the article suggest several cycles of manual evaluation and tuning before there's even a test involving searchers in the wild.

With regard to many justifiable concerns that Leosghost raises, I'm conjecturing that many of the tweaks that irritate us all may in fact be "controls" designed to filter out certain kinds of biases. Perhaps that testing and retesting accounts for many of the apparently cyclic results we've been talking about in our various Zombie traffic threads and elsewhere...

Zombie Traffic and Traffic Shaping - Analysis
http://www.webmasterworld.com/google/4437835.htm [webmasterworld.com]

[diberry]

The problem with the blind taste test approach - when both flavors are coming from your own brand - is that one of the two options has to win. There's no third option as a control to indicate whether you're on the right track at all.

Why not give them three sets of serps - one pre-Penguin, one post-Penguin, and one put together by a human using the top 10 pages from both sets of results - and ask them to grade all three according to certain factors? If pre-Penguin got the highest grade from the raters as a whole, you'd know your Penguin algo needed more work. If post-Penguin won, it's time to break out the champagne. But if the human one got it, that could indicate problems with your overall approach.

It's possible Google already conducted a test like that somewhere in the process and we're just not hearing about it. But if not, then it sounds like they're not even testing for the possibility that the algo is just plain off course.

I don't expect every algo tweak to be perfect right out of the gate, but Penguin is delivering the strangest results since a few years ago (was it 2005? 2007? I think this was what preceded Florida, but not sure) when the spammers were totally beating the algo on pretty much every query, and the entire first fold was usually worthless.

[Whitey]

Singhal was adamant: “no revenue measurement is included in our evaluation of a rankings change.” Listening to him explain how excites he gets about search improvements and how changes are evaluated, you realize there’s no spin here. He’s absolutely telling the truth. And he would know. Chris Sherman asked if anyone at Google really understands how the whole thing works and he replied that while no one knows how everything works (all of unpaid search, AdWords, Android, etc.), he has a pretty good idea of how all of unpaid search works. Not many can make that claim.

Search results may be exonorated, but it's often not relevant in the context of the page layout.

There would have to be considerations as to what get's placed on a page. I mean , Adwords , Maps , Place etc. Do we ever hear from the page layout team ?

[rlange]

My only concern with this type of testing in Google's situation is that two separate changes, tested in parallel, may each improve the results, but the combination of the two may have a negative effect. You can use a number of spices, individually, to improve the taste of your food, but you probably don't want to use all of them. Implementing a good update before testing the next one is the only way Google could get around that, but with as many changes as they test each year, it's impossible.

I'm likely not nearly as intelligent or experienced as the folks working at Google, so I have to assume that someone, at some point in the company's history, had the same concern and that it was either addressed or determined to be insignificant.

Leosghost wrote:
@rlange..I was replying to the contents of your post prior to your editing of it [...]

Yeah, sorry about that. It's a bad habit of mine.

--
Ryan

[Planet13]

One amongst many flaws in Google's approach is that Google have no way to be sure ( other than asking the "lab rats" ) if they have ever run a search for the terms whose results they are given to assess and assign a preference to..

That is a good point.

I also wonder whether the human testers are made up of a broad section of diverse backgrounds. After all, preference is subjective.

When my wife was looking for a treatment for a particular medical condition, she was dismayed at the lack of acupuncture and holistic remedies listed, since the top SERPs were dominated by western medical sites. Her preference to Eastern / holistic treatments is probably due to her ethnic background.

So it would be interesting to get more info about the background demographics of those who help rate the results. A native San Franciscan would have different preferences to, say, a native Georgian.

[diberry]

@rlange "My only concern with this type of testing in Google's situation is that two separate changes, tested in parallel, may each improve the results, but the combination of the two may have a negative effect."

Very good point. And I agree that someone has surely had the brains to bring it up at Google.

The problem in big business, however, is not lack of intelligence but the distortion of emotion. I've seen medium to large companies decide they're simply not going to test for a particular risk because the answer might upset stockholders or indicate the company needs to start over from scratch, etc., and since no one wants those answers, they just don't do the tests. Take the banking industry: surely plenty of people in that industry knew the upside-down mortgages and credit default swaps were going to implode like a Ponzi scheme eventually, but they were making so much money, no one wanted to hear that. So they didn't listen.

Again, I'm not at all saying that is what happens at Google - we can't be sure from this snippet that Singhal shared the whole testing process. I'm only saying they wouldn't be the first huge company to limit their tests according to company policies rather than logic.