New Interview with Google Engineers about Algo

[aristotle]

The interview can be found at

[msn.finance.com.my ]

Here is a quote:

Proposed changes to Google's formula are first tested on a separate set of computers that imitate real-world search.

Those deemed worthy are next sent to evaluators around the world who act as online searchers and rate the relevance of results in various languages and regions.

Google then does live testing, with promising algorithm enhancements carefully blended into results served up by the main search engine.

"At any given time, some percentage of our users is actually seeing experiments," Huffman said.

[Reno]

There is nothing they do to support their claims of "positive", "successful" or whatever.

I've been suspicious of those claims from the beginning. I mean, how many average websurfers stop to find the Google support email and take the time to say "Dear Google, I am very happy with your new results".

Answer: Zero, so I have to assume they get their feedback from the "evaluators around the world".

We know of course that Panda was at first only rolled out in the USA, so the feedback would presumably be from USA evaluators only. What we don't know is anything about the demographics of these people ~ we don't know the total number of evaluators, we don't know their level of search sophistication, or their test queries, or their criteria for "success". For all we know, the fact that they got ANY results may have been enough to make them happy.

All of which makes me suspicious that the official comments, metaphors and folk tales are mostly good PR ~ after all, they've got a full year of work by a bunch of high paid engineers, so it would only make sense that they're going to put a positive spin on this thing, no matter how ugly it is.

............................

[tedster]

They might be getting positive input from profession research people and reporters.

[Reno]

They might be getting positive input from profession research people and reporters.

Both of whom may be what they mean by "evaluators". Input from those folks may be a useful micro sample, but I would guess that neither of those groups is deeply representative of the average websurfer at one end of the scale, or webmasters at the other.

....................

[Shaddows]

If we make a change to our site, I know whether it has been successful or not. And we never ask customers about it (apart from indirectly through user satisfaction enquiries).

[indyank]

But will you still claim that you have got an "overwhelmingly positive feedback"?

What is this feedback? Is it end user feedback or their search quality team's feedback or something else?

If 20% of searches were done by webmasters (which I don't believe) and if this feedback was "end users" feedback, then the feedback they have received is not at all relevant.

If it is their experts' feedback, it is also not totally relevant as the expert population doing searches on google will be a very small percentage and it can definitely not be "overwhelming".

[aristotle]

According to what I've read, Google has about 10,000 full-time employees, mostly in India, whose job is to check and evaluate the SERPs, and also to evaluate individual websites.

I don't know the details of what these 10,000 employees actually do. But it has occurred to me that collectively they could check a lot of individual sites. For example, an employee who spends an average of 10 minutes on each site could check 6 sites an hour, or 48 sites in an eight-hour day. So altogether these 10,000 employees could check 480,000 sites per day, or more than 3 million per week. Most likely Google would assign them to check the sites that get the most traffic. Of course this is just specualtion on my part.

[danimalSK]

don't know the details of what these 10,000 employees actually do

Google's instruction sheet for external raters leaked a couple of years ago, see here:

[shenouda.nl...]

[Shaddows]

But will you still claim that you have got an "overwhelmingly positive feedback"?

What is this feedback? Is it end user feedback or their search quality team's feedback or something else?

Good questions.

The only way they could get high-volume, credible feedback is through a semi-autonomous "focus group". I'm thinking payrolled quality assesment types, who don't do any actual algo work. Normal-ish people who rate whether something meets independant criteria- like the ones they used in creating Panda.

Now, in a big company, the algo team would legitimately treat non-algo types as "other people"- and would only see collated data, not human responses. And to anticipate one objection- the "focus group rated" searches would come from a centrally compiled list of most popular searches, or as a random sample of live searches from real users.

In addition, they have automatic measurement of SERP satisfaction in place for all searches- presumably they are happy with resultant trendlines. Data-led types would be perfectly at home stating an up-ticked trace on a defined metric was "positively received" or similar.

Added- posted at same time, but my "focus group" would be danimalSK's external raters

[rlange]

Dan01 wrote:
They worked on it for a year.

Here's the thing, though:

Amit Singhal: Well, we named it internally after an engineer, and his name is Panda. So internally we called a big Panda. He was one of the key guys. He basically came up with the breakthrough a few months back that made it possible.

Source: TED 2011: The ‘Panda’ That Hates Farms: A Q&A With Google’s Top Search Engineers [wired.com]

That article is dated March 3. Granted that "a few" is non-specific, but for the man for which the update was named to have only made the breakthrough "a few" months before it was rolled out, how much and how thorough a testing could it really have received?

--
Ryan

[Reno]

Good sourcing Ryan. I sure wouldn't want a disaster like "Panda" named after me ~ too bad the guy's name wasn't Anthrax.

..................

[rlange]

Reno wrote:
I sure wouldn't want a disaster like "Panda" named after me [...]

To be fair, I doubt that the Panda update is the wide-ranging disaster that some of us are making it out to be. It would be insane for Google to proceed with the international update if it were and I still have at least that much faith in them.

--
Ryan

[Reno]

and I still have at least that much faith in them

You have more faith than me ~ the evidence speaks for itself, and to my eyes, the evidence is that a gigantic update went awry for many millions of siteowners that had followed the guidelines religiously. And others, that did NOT follow the guidelines have been rewarded. Something is very wrong with that picture.

I'd like to think they see that, but the press releases clearly indicate that they are feeling mighty good about everything. Maybe they're working behind the scenes ~ we'll see.

.......................

[tedster]

press releases clearly indicate that they are feeling mighty good about everything. Maybe they're working behind the scenes

Whatever they did between Panda 1 and Panda 2 managed to put a dent in eHow, at least according to the Sistrix data. I'd say they certainly will continue to work on this. Matt Cutts mentioned elsewhere that the quality algorithm will be one of Google's year-long focuses for 2011.

It is not easy to analyze the overall effect and know whether it really made an overall improvement or not.

We know that Panda negatively affected a lot of websites - possibly more than any other update ever did. That means there will be many more complaints from site owners. And those site owners will scrutinize the particular SERPs that are important to them, looking for flaws, and those flaws are there.

But the SERPs had many flaws before Panda, too. So did the new algorithm generate an overall improvement? How can we begin to answer that question, unless we have all of Google's data?

[crobb305]

deleted my comment. Wrong thread.

[TheMadScientist]

But the SERPs had many flaws before Panda, too. So did the new algorithm generate an overall improvement? How can we begin to answer that question, unless we have all of Google's data?

Simple, they didn't revert it.

Somehow, I'm sure a company with the storage, processing power and sheer size of Google has the ability to revert a change if they need to, even one this large, but instead of going back they refined. If Panda had really made things worse, they would have pulled it, imo.

[mrguy]

Matt Cutts mentioned elsewhere that the quality algorithm will be one of Google's year-long focuses for 2011.

Taking that long used to not be a problem because Bing was only barely eating into the share. If Google continues to lose percentage of share like last month 3%, it becomes a problem real quick because in 6 months at that rate they will only be at 50% share in the US.

If Facebook jumps into the fray like they should, after all Google is gunning for them as well, then Google's problem just got much bigger.

The little quirky search engine those of us who have been around a while remember is gone forever and it will be interesting to see what the next year brings.

[TheMadScientist]

A bit more on the 'they didn't revert it' line or reasoning ... Remember how many times they jumped to and from Caffeine? That was an infrastructure change where the two different infrastructures (Big Daddy & Caffeine) weren't compatible, and they managed to show it 'off and on' for months ... I'm almost certain they could and would have reverted Panda if the data showed they didn't 'get it right' from a big picture perspective when it launched.

[walkman]

Somehow, I'm sure a company with the storage, processing power and sheer size of Google has the ability to revert a change if they need to, even one this large, but instead of going back they refined. If Panda had really made things worse, they would have pulled it, imo.

All we need to know that this algo is #$%^& up: Not one site has come back up on its own, as far as we can tell, almost 2 months after the change. And we know innocent sites hit hard.

[tedster]

The absence of recovery is a very disturbing factor. I've been wondering if it's an intentional move, to keep people from a disciplined reverse-engineering of the Panda factors. It's hard to believe that some of the extensive site changes people are talking about making had no effect.

Well, in some cases people are reporting even lower traffic, but only a handful are reporting any improvements, and maybe three total are claiming complete recovery. That's such a small number that I wonder if they were really hit by Panda at all. At any rate, it's too small to be statistivally significant.

If Panda re-evaluations are really "on hold" in some way, some communication from Gogole about that basic fact would be a good thing.

[supercyberbob]

I'm waiting for the "algo change" in the brain cells of search engine users world-wide that drops traffic to Google.

Like, Panda, it will be irreversible and permanent. :)

AND, we won't be making any manual exceptions.

[TheMadScientist]

I've been wondering if it's an intentional move, to keep people from a disciplined reverse-engineering of the Panda factors.

I think it well could be, and I posted in another thread I would not recommend the usual 'one-at-a-time, figure things out' approach to changes for this one, but rather 'new site' or 'new focus' oriented changes on a more broad scale.

If Panda re-evaluations are really "on hold" in some way, some communication from Gogole about that basic fact would be a good thing.

I think we already got that via John Mu's comments I got into a discussion about for the 'over-analyzation' of the comments ... He said, 'can and will' take time for recovery ... I'm getting tired of posting on this subject and getting 'beat up' by those who just want to say 'Google's broken', so I think I'm going to try and take a break from it for a while ... It gets old.

[walkman]

"He said, 'can and will' take time for recovery"

"Time" started 13.7 or billion years ago and we don't know when it will end. And in time we all will die

[Reno]

some communication from Google about that basic fact

This is the very essence of my argument & complaints about Google. Their deeply offensive "Fear, Uncertainty, Doubt" policy (FUD) is at the root of so many of the problems we all have with them, especially now with the deadly Panda on the loose. The sad truth is, clarity in their communications could have helped to mitigate many of the issues we are now discussing.

If they had come out 6 months in advance and said something like the following, we'd have had the opportunity to brace ourselves:

"We're putting everyone on notice that quality will be a major facet of our next algorithm update. FYI, we define quality as:
- Original content;
- Natural language;
- Natural linking;
- Clear navigation;
- Valid coding;
- Fast download times;
- No blackhat SEO..."
(etc etc etc - you get the picture).

Instead, they dropped Panda on siteowners like a drone attack, and it wiped out many innocent people. With little or no advanced communications, the collateral damage on their part can only be seen, IMO, as intentional, and puts the final bullet through the heart of their so-called "Do NO Evil" corporate philosophy (if that was ever true!).

..................

[tedster]

"Do NO Evil" corporate philosophy

I think it's actually "Don't BE Evil." Google has always known that their actions can have damaging effects on other businesses, and those businesses naturally see this as "doing" evil. Unfortunately, any big business - not just Google - does create effects than are felt as evil.

In Google's world view, they don't intend evil. It just sort of comes along with the territory. And I have some sympathy for that - I don't think they intentionally "do" evil at all. But I also don't think they ever had even an informal definition for "evil".

[walkman]

If they had come out 6 months in advance and said something like the following, we'd have had the opportunity to brace ourselves:

"We're putting everyone on notice that quality will be a major facet of our next algorithm update. FYI, we define quality as:
- Original content;
- Natural language;
- Natural linking;
- Clear navigation;
- Valid coding;
- Fast download times;
- No blackhat SEO..."
(etc etc etc - you get the picture).

Instead, they dropped Panda on siteowners like a drone attack, and it wiped out many innocent people. With little or no advanced communications, the collateral damage on their part can only be seen, IMO, as intentional, and puts the final bullet through the heart of their so-called "Do NO Evil" corporate philosophy (if that was ever true!).

That would have been the right thing to do, but it would have taken Matt Cutts 5 minutes to write and maybe he felt that he would have lost his aura a bit. Of course it could be worse: like losing your Gmail access and with that Adsense, Blogger, Picasa and their affiliate network access...and getting zero support from them. It's going to be fun when they ask people to put everything in Google Cloud. It's not related to Panda but part of their attitude and frankly arrogance.

Obviously I have no choice but to keep my site in Google but for the rest, I consider them lost. At any minute for absolutely no legit reason they can cut them off for me or for anyone and maybe get an email to tell me there's nothing I can do do about it.

[Reno]

In Google's world view, they don't intend evil. It just sort of comes along with the territory.

A. The road to hell is paved with good intentions;
and
B. Yes, any algo change is going to shake things up by definition, and there will be gains & losses from that shift. Again, that's not my argument. I'm saying that we are LONG past the days when they had to be careful about every word they uttered, for fear that a couple of other guys working late in the library would torpedo the company. They are now the Roman Empire of search ~ they can afford to give the townspeople notice before they burn the place down. At this point it's a matter of decency.

....................

[tedster]

They are now the Roman Empire of search ~ they can afford to give the townspeople notice before they burn the place down.

Well said. I don't know what exactly "straight" webmasters could have done, but I guess some advance notice might have softened the blow a bit.

[viggen]

They are now the Roman Empire of search ~ they can afford to give the townspeople notice before they burn the place down.

The question would be then what Roman Empire, the one of the Punic Wars, the Fall of the Republic, the Five Good Emperors, or the decline and fall?

...some people mentioned that no one recovered so far from Panda, i also believe that this is on purpose and might stay that way for 6 months or so, would be around the threshold to throw away websites that only made money (MFA). If so it would be a brutal step, but in the end the little guy that does it not only for money but also because he invested lots of time, energy,love and passion would stick it out, while the thin content spinner would move on?

just wondering...

[Dan01]

MrGuy:

If Facebook jumps into the fray like they should, after all Google is gunning for them as well, then Google's problem just got much bigger.

I wonder if Facebook would jump into search. They would be sending users to other sites. Other sites where other ads appear. I think their goal is to keep you on their site so you can see their ads.

Now that can change.

I would love to see their ad volume increase so I can put their ads on my sites.

Speaking of that. Mediaweek had an article about CTR and types of sites. Facebook had one of the lowest CTRs.

[tedster]

Three weeks ago, Facebook sent this official response to a SearchEngineLand, after an article conjectured that they would get into web search based on some screen captures that apparently showed a web search test (see our thread for more [webmasterworld.com] )

We are not testing the placement of a separate web search field and have no plans to do so. We believe the second search field or "Search the Web" box appeared on peoples' accounts as the result of unknown actions by a third party targeting the browser (potentially a browser plugin or malware) unrelated to Facebook.

[searchengineland.com...]