Forum Moderators: Robert Charlton & goodroi
Nothing earth shaking, but I enjoyed this look at Google's internal "Replay" tool, that helps them back up their algo to see what happened a little while back:
It was getting hard for engineers to [prove] something happened on Google. Other engineers would read their email 10 minutes later, and say, it's working fine for me.So they developed a tool called Replay where they can freeze time retrospectively and take a picture of it. I can say, What would have happened had I run this query at 12:49? Then we can go debug all the systems and say, Ah, that system failed.
[businessweek.com...]
"Both using our internal raters, human beings, who say, yeah, good thing, and observing real users in a tiny fraction of our traffic, observing how they interact with something. And we have metrics, like clickthroughs."
He mentions both "internal raters, human beings" and "metrics, like clickthroughs". But I'm wondering if these are just limited to "experiments", or are they also used as part of the real rankings process.
He mentions both "internal raters, human beings" and "metrics, like clickthroughs". But I'm wondering if these are just limited to "experiments", or are they also used as part of the real rankings process.
I gather from Matt's interview that these are likely to be used to identify problems. Matt in fact refers to what I assume are 'clickthroughs'....
If someone did 15 queries in a row and never clicked on the results and eventually left, that may be the sort of thing where you dig in and say, well, did we have horrible results? Were they looking for a picture and we never returned a picture?There’s a lot of different ways we gather all that data to identify a problem. Once you’ve identified a problem, that’s when the fun starts, because you can brainstorm a little bit.
This might be where that Replay tool comes in.
Reading both interviews together gives a good sense of the process.
The interviews (there are four in all) reveal very little. Google is very good at not saying anything.
Cheers
Sid
I also noticed that from Friday October 16th from around noon rankings just changed...
Whether or not Google fizes Friday's shift by using the Replay Tool that Amit Singhal mentioned in the interview, we're not likely to know. If anyone has more observations about changes in the SERP, the best place to share them is this thread: Google Updates and SERP Changes - October 2009 [webmasterworld.com].
In addition to the article's title quote ("We Will Try Outlandish Ideas"), I also thought this comment was interesting - it seems to describe the need for Caffeine:
Q: Udi talked about the search quality team doing more than 6,000 experiments a year. How does Google do that many in a year?A: You need that infrastructure where they can go from idea to data in a few hours.
Also this quote. It may just be a good PR-value statement, but it also sounds to me like Amit has the attitude that he describes:
Q: Is there anything you do to make sure you don't get in a rut?A: That's the biggest dilemma. That's what I tell my team. What we may have is a very well-oiled oval wheel. And when the first round wheel comes along....
I would be lying if I said I had a magic bullet to solve that. It's the openness to innovation, the desire to dismantle status quo.
Sorry if I took the thread off at a tangent.
I was interested to read the bit about stemming and the first attempt not working. Sounded like the Florida update to me!
I wonder if this guy is responsible for all of the Google cock ups of the last few years. I gathered from what he says that they are prepared to do trial and error on an ongoing basis on the live data.
Cheers
Sid
But the article kind of blurred the line between Udi Mamber testing interface layout changes (6000 experiments in a year) and algorithm updates (over 400 in a year).
