Fate of small sites in Google's SERPs: it all depends on statistics

[econman]

In the middle of a long, interesting Hacker News discussion concerning changes and trends in Google's search results, I noticed a couple of very interesting comments from Moultano, a Google Employee.

https://news.ycombinator.com/item?id=6572688 [news.ycombinator.com]

Google does not make ranking changes based on revenue. We don't even collect statistics about that for ranking changes. Worrying about the fate of small sites in Google's rankings is a reasonable concern, and one that we share. But whatever you think we are doing incorrectly in ranking, it has nothing to do with making more money.

In a subsequent post he says:

I know you're basically just trolling, but I'll reply seriously anyways.

I'm one of the people who makes changes to Google's rankings. As a result, for any change I want to make, I have to collect the statistics to justify it. This is done with the help of an analyst who has a different reporting chain from the ranking engineers in order to ensure that they remain unbiased. These statistics include things like what results people click on, how often people hit "next page," how humans rate the results before and after the change, etc. (None of these statistics involve ads or revenue in any way.) Once we've collected those statistics, the analyst writes up a report about the change summarizing their findings and pointing out any areas of concern.
This report is then presented at a weekly launch meeting, where the ranking leads review each change both for its metrics and for its complexity, ongoing infrastructure cost, etc. and make a decision about whether to launch it.

The immediate thrust of his comments is simply to defend Google by explaining about Google's "Chinese wall" between the folks who control the algorithm and the folks who work on monetizing the SERPs (including the ones who decide whether to push the organic results far down the page).

And, I suppose some might argue there is nothing new here. But his comments struck me as well worth thinking about and discussing more deeply because his comments provide a glimmering of insight into how Google functions as an extremely data-driven, engineering-dominated organization, in which the focus on data and statistics permeates the culture from top to bottom.

I also found his specific examples interesting, including their reliance on data concerning how many users hit "Next Page," and how human testers rate the results of various experiments.

[edited by: aakk9999 at 7:13 pm (utc) on Oct 21, 2013]
[edit reason] Added link to Hacker News thread [/edit]

[Panthro]

Nice find

[brotherhood of LAN]

Interesting

>These statistics include things like what results people click on, how often people hit "next page,"

I think those stats may get a little noisier! I guess it depends on what Google qualify as 'people'.

>how humans rate the results before and after the change, etc

I'd assume they mean their own humans who do quality scoring?

>(None of these statistics involve ads or revenue in any way.)
>review each change both for its metrics and for its complexity, ongoing infrastructure cost, etc

You could argue the 1st line contradicts the 2nd, but in general it does seem like a very forthright answer.

[econman]

I'd assume they mean their own humans who do quality scoring?

That's how I took it. Reading this, I visualize (just guessing, of course) a process in which engineers are free to think up potential "improvements" then test their ideas. For instance, data for A:B comparisons between their idea and the existing algorithm can be generated by switching to their experimental SERP results on 1% of the "live" queries by regular users, and on 10% of the results fed to the people Google pays to "rate" various query results.

Neither the regular users nor the paid result raters would have any way of knowing whether they are looking at results of the regular algorithm, or the results of an experiment, or even a combination of the two -- but the system will keep track of exactly what happens each time. After enough data is collected, it can be analyzed in great detail, allowing the engineers to study whether the experiment achieved the expected improvement -- as well look for unintended consequences.

[SevenCubed]

So where are the engineers that worked at Google in 2008 now working?

[JVerstry]

Very interesting.

I am more and more convinced that Hummingbird has some (many?) undesired side-effects that Google still needs to fix. Sept 4th coincides with many sites going down without any explanation. This is also when people started to complain about search result quality.

I don't think it is an accident if Google started to ask about "small websites which should rank better" at the end of august, right before Hummingbird's release. Why would they ask for feedback if it was not a known issue?

I think they have decided to push ahead with changes anyway, and it is backfiring in the webmaster community. I also believe there are many side effects they are only discovering now.

I also believe they are about to learn that representations of reality (be it algorithmic) is not reality itself. Reality is also changing faster than statistical models do. Models may be temporarily or locally correct at best. Who can predict where fashion, a very human activity, is heading with statistics?

Running the same matrix over and over as an Oracle produces its own biais not matter what. Reality always escapes reprsentations on the long run, always.

I also think:

1) they may learn the hard way that all business issues cannot be solved with scalable algorithms only. Their model did not predict the current level of upset in the community.

2) they should also start to make the difference between controlling and collaborative relationships.

3) they don't understand that disempowering spammers does not empower honest webmasters automatically

4) they ought to understand that hidding information to fight spammers is also preventing honest webmasters from learning from their mistakes, or simply learn what Google wants or need from us

5) they should provide explicit feedback on website quality issues in GWT (thin content, dups, low quality, etc. everything), because telling spammers they are doing something wrong is not something they already know. And if they fix quality issues with their sites, what is wrong with that?

But the bigger benefit is that honest webmaster will use this feedback to develop better sites and users will follow them, not spammers. Google should trust this process. This is what I mean by the difference between controlling and collaborative relationships. Right now the system is too anal.

6) Google needs to understand that frustration, silent penalities etc. leads to lack of motivation, which leads to less new valuable stuff being created, which means less quality stuff to serve to users. Right now, this side of the equation is not handled properly at all. Many people feel like being treated like cattle.

7) If a layer of spammy crap is removed, it does not mean there is necessary gold underneath it. Usually, there is only more crap. In other words, putting all the money on fighting spammers does not create gold. It is only trying to suck more from a depleting mine. There has to be a better return and treatment for those creating gold.

8) Google should stop using unpaid, unqualified and uninformed people to do the job their engineers should do on their forums. The quality of answers, when there are answers, is too low, and often useless.

People are often coming back, over and over, with the same questions. This does not happen on stackoverflow, why? Tip: it is about collaborative relationships and sharing.


Overall, I also think Google is short of a Senior Webmaster Engagement Business Manager.

Someone:
- who does not necessarily have a technical background
- who has been facing the customer, I mean NOT behind a screen only
- who has good listening skill
- who does not go defensive or takes things personally in the presence of criticism
- who can act as a customer advocate for Google own's interests
- who understand the value of a proper relationship with suppliers
- who is properly empowered to push key business decisions

Often, businesses need to change gear. I think it about time for Google. Google needs to own this one.