Does Google Whitelist Certain Websites?

[tedster]

Webmasters have long wondered if Google maintains a "whitelist" that protects certain domains from penalties. As reported on SERoundtable today, Google issued a statement to clarify the question. Their answer denies anything like a "get out of jail free" card, but it explains what they call "exception lists".

Like other search engines (including Microsoft's Bing), we also use exception lists when specific algorithms inadvertently impact websites, and when we believe an exception list will significantly improve search quality. We don't keep a master list protecting certain sites from all changes to our algorithms.

The most common manual exceptions we make are for sites that get caught by SafeSearch - a tool that gives people a way to filter adult content from their results. For example, "essex.edu" was incorrectly flagged by our SafeSearch algorithms because it contains the word "sex." On the rare occasions we make manual exceptions, we go to great lengths to apply our quality standards and guidelines fairly to all websites.

[seroundtable.com...]

[netmeg]

So the answer is pretty much - "yes"

[SanDiegoFreelance]

I would assume a number of them. SafeSearch would be one of them.

Since they are constantly working on the algos almost necessarily each algo would need a test list starting with a seed list. They would also need a white and black list to test against.

The panda list is likely their latest list? Maybe a panda exception list is in the works?

[tedster]

For Panda, my guess would be that they are building a preliminary list of false positives. But that will not followed immediately by an exception list. Instead, they will modify the algorithm so that kind of site doesn't get caught by it. Matt Cutts did say recently that site quality will be Google's focus for this entire year,

I'm remembering an incident from 2009 where a site owner reported a ranking problem on the Google Webmaster Help forums. In the end, JohnMu found that the part of the algo that simulates the visual page was flagging his site as having "too much whitespace" - something that spammers apparently had.

John said he "set a flag" so that any time the algo picked out his site for that issue in the future, it would let him pass, but trigger a human review instead. I think that's an example of how Google will whitelist a site - just for some specific algorithmic property, rather than giving a free pass to do whatever they want.

[Simsi]

A question on this whitespace thing Tedster: I manage a small network of sites in various locations that pull in panels of content from a centralised domain database. It uses iFrames to do this. This content is created on the fly and very relevant to each site but consequently needs to be centrally managed.

In the new preview tool on G, where you can see an image of each site, these iFrames show as big panels of whitespace. I have as yet seen no ranking oddities after 4-5 months of having them in place, but could this potentially be an accident waiting to happen based on the above observation?

[tedster]

In the nearly three years since that incident, I have heard of no other cases. At the time, JohnMu did say that it was a very unusual combination of factors that caused the false positive. What you are describing is a MUCH more common event, by comparison. If I were you, I'd have no worries.

----

Here's what I'm wondering about. Given the recent information that reconsideration requests cannot reverse and algorithmic penalty, I wonder if requests are at least screened for false positives.

[TheMadScientist]

<?php
$NoIframes = file_get_contents('http://example.com/the-site's-external-content-location.php');
echo $NoIframes;
?>

[CainIV]

Like other search engines (including Microsoft's Bing), we also use exception lists when specific algorithms inadvertently impact websites, and when we believe an exception list will significantly improve search quality. We don't keep a master list protecting certain sites from all changes to our algorithms.

To me that is the exact same thing as saying:

Yes, under some conditions we do, and will, manually change results to impact our perceived vision of improved results.

[BeeDeeDubbleU]

... but don't think wrong of us for doing so because Bada Bada Bing does it too. :)

[anand84]

I have long believed a whitelist exists that includes a number of major publications. One reason I think so is because a number of such websites are not "penalized" for duplicate content (unlike lesser mortals like us) from sources like Reuters, AP,etc. (which is carried by a number of these big publications).

Ideally, in case of duplication, Google should only rank the original Reuters news article in their SERPS. However, it is not uncommon to see the same article from other sources like NYTimes appear on the Google search results as well.

[TheMadScientist]

John said he "set a flag" so that any time the algo picked out his site for that issue in the future, it would let him pass, but trigger a human review instead.

I would think that statement could be a bit unnerving for some people who might think being on an 'exception list' would be a 'good thing' ... A manual review of your site may get triggered every time you trip the same filter you tripped in the 1st place?

Do you really want to be on that list?
Really, stop and think about it for a minute...

I hope your server is up and running all the time, you don't have many (if any) broken links, any 3rd party sites you rely on for information are super fast and responsive, you have high quality content with great presentation of information, and you don't do anything at all shady, because they say they only add a list when they feel it 'will significantly improve search quality', and my guess is to be on that list your site has to be really close to their definition of quality, so I personally wouldn't want to give them any hesitation in keeping the site listed.

That's not a 'white list' that's a list for additional, repetitive scrutinization...

A 'white list' is when you 'blindly include' a site without a thought or a glance at it, you don't go manually review sites on a 'white list' to make sure they're really the quality you want to include, you include them regardless of what they do, and what's unfortunate is another SEO myth is likely born, even though an 'exception list' is likely a list many webmasters probably don't really want to be on.

[tedster]

Today, SearchEngineLand published an exact transcript of this whitelist discussion at SMX. Up to now we had a partial paraphrase. Interesting to me was the final comment from Matt Cutts:

Cutts: Yea, it is also important to realize that there are many many algorithms, maybe the majority of the algorithms that don’t have any exception logs. For example, Panda, we don't have, there is not way right now, to do any manual exceptions.

Sullivan: No, no, I read the Cult Of Mac you'd exempted things for them. [sarcasm]

Cutts: Nope

[searchengineland.com...]

The article also has a video of this part of the session. Unfortunately the visual part is just the session slide. But you do get to hear the voices and that gives other signals that a mere transcript can't convey. We also get a new piece of insider language from Google: "exception log."

[bwnbwn]

I know of a website that has the non www version linked, the www version linked and the ip linked. All display a 200 and has never suffered. We all know what indexing 3 versions of a site will do and this method is used to take a site down, but the site has never been effected. I am not the one doing the linking not my game but I know who is and have seen many links pointing to all the different versions from a number of sites on different servers, Ips, and regions.

[tedster]

Having canonical problems like that is a bit of a different thing from whitelists, isn't it?

I have noticed in the past year or so that for many sites, Google seems to "fix" canonical errors on their own back end. This is often the kind of thing that would have tanked a site badly in the past. I'm not sure why they can only do it sometimes and not others - maybe it's related to consistent internal linking.

So I wouldn't consider this to be a kind of whitelist. Canonical URL problems are usually a technical thing, and not really an algorithm factor, per se. Google's has been accommodating more of the common IIS nonsense too. Things like 302 > 200 for not found errors are often not the big issue that they once were.

[bwnbwn]

Having canonical problems like that is a bit of a different thing from whitelists, isn't it?

Ted is it? I am not so sure.

maybe it's related to consistent internal linking.

The whole site diplayes in ip, non and www. If this were to happen to a site of mine today it would be nuked. There has to be something that prevents this site being effected. I have watched this site go through all the Google updates nothing has ever effected it. I have for along time suspected a whitelist. I might add the links were added 3-4 years ago so if it was a past problem it didn't work then either.

[tedster]

To my thinking technical issues are a whole other category of SEO compared to the guidelines about what gets penalized. You can have major problems with rankings from technical errors, but it's not a penalty - no demotion is involved and there's no "black mark" on your record. It's more about missed opportunity, diluting link equity and things like that,

[wheel]

The only people who don't want to think Google doesn't make hand exceptions, or lists all the time, is Google. And that's just so they can screw people and still claim to be impartial. For the rest of the world, of COURSE they whitelist - I doubt you can run a decent SE without doing so.

. For example, "essex.edu" was incorrectly flagged by our SafeSearch algorithms because it contains the word "sex."

If I owned essex.edu, I would totally be the king of pr#n right now.

[bwnbwn]

O sorry Ted I missed the penality part in your post yes I agree with you I was referring to some "preferred" sites in general seem immune to most issues we as normal sites face.

[aristotle]

Instead of a whitelist, some websites might have acquired a high trustrank or authority status that overcomes other factors that could cause a lesser site to fall in the rankings.

[indyank]

Even if they maintain algorithmwise exceptions list and not a masterlist of whitelisted sites, they might still include the whitelisted sites in all those exceptions list.

Panda update is exposing these whitelisted sites (or sites that figure in most exceptions list) more clearly now.

If someone thinks that whatever these top rated sites do is nothing wrong (as they still rank on top) and try to do the same to their sites, they will be shooting themselves down.

By maintaining these exception lists and floating them on top, google is setting bad examples for others.Sure, these sites may be good and useful to the readers but most things they do to remain on top may not be good examples for others to follow.

[BillyS]

Safesearch is a poor example of whitelisting since the parameter can be changed by the user.

In any event, I would think this approach would be used sparingly since the downstream implications can be quite unpredictable.

[tedster]

Yep.

Cutts: Yea, it is also important to realize that there are many many algorithms, maybe the majority of the algorithms that don't have any exception logs.

And even though this comment came from Bing's Sasi Parthasarathy, I'm pretty sure it would apply to Google's process, too: "But the interesting part is that any time we have these static overrides, we will make sure that the next iteration of the algorithm actually handles these lists. So these lists are constantly evolving."

-----

Here's how I envision it working. In the early days of the hidden text algorithm, sites used to trip a false positive because the page had a white background but the text container had an image background. They logged the exceptions and then upgraded up the algorithm when they had the data and a approach that worked.

[Whitey]

the majority of the algorithms that don’t have any exception logs. For example, Panda, we don't have, there is not way right now, to do any manual exceptions.

To me , most QA at Google practices boil down to a common sense approach to a problem which for all practical purposes should be obvious to webmasters. The difficulty lies in how Google administers that common sense, if indeed it is at all able of doing so.

There appears to be capability in these areas :

> algorthmically adjust
> algorthmically log
> algortmically plot patterns from logs
> algorthimcally flag
> user flag
> algorthmically penalise or rerank
> editorially review
> editorially log
> editorially provide white or black status

I wonder how JCPenny and others got through this until recently though. Maybe the system has some flaws.

This is just my perception - others may take a flamethrower to my remarks.

[mromero]

tedster - Last year it was reported G is collaborating with the NSA.

On one niche we monitor the CIA and State Dept. are locked tight in their positions in the SERPS no matter how irrelevant, outdated or silly the results are.

One page has data from 2006 but still promoted by G as latest info - simply to my mind because the C.I.A. sez so.

Now leading even wikipedia.

Smells like a whitelist....

[tedster]

Have you checked backlinks for those pages? When a domain has accumulated trust and authority, "outdated" or "silly" is exactly the kind of thing an algo would rank.

[frontpage]

I think the claim that Google results are natural based on algorithms is pretty much debunked.

Google has stated that they hand manipulate results and that includes placing their own links at the top of SERPS.

This will be further weight to the government charges of a monopoly that is abusing its position.

The European Commission in November started investigating whetherGoogle is illegally directing users of its search engine to websites it owns or is affiliated with and whether Google is stopping websites from accepting competitors' ads.

[edited by: frontpage at 12:41 pm (utc) on Mar 17, 2011]

[almighty monkey]

I guess the existance of an exception list makes sense.

Say, there is a big major news story, like the Japan quake. The BBC put up the story and a million bloggers, forum posts and twitterers link to it in an hour.

Without a whitelist, Google would just go 'AH SPAM!' and dump them from the SERPS.

[TheMadScientist]

<OffTopic>
It's amazing to me how the sentiment about Google has changed right here in these forums over the last few years ... I really think the turning of webmasters away from Google is probably the beginning of their decline back to 'reasonable market share', because we have a HUGE amount of word-of-mouth reach and influence as a group, and my guess is some are starting to use it ... I think one of the biggest mistakes they've made along the way is for their rep(s) to stop posting here ... Anyone else remember GoogleGuy? I used to think other reps that 'copied' or 'mirrored' the his user name were completely lame, but now I'm happy to hear from any of them ... When they dropped webmasterworld.com from their 'posting list' it took away all the "we're hearing you" feeling we used to get when something changed and we didn't understand why, because even if they didn't fix everything or reply to every post, at least they took the time to try to explain and even gave us a way to provide feedback on more than one occasion, but now we rarely get even a courtesy 'fly by' ... Even though I didn't get hit by the Panda Farmer Update and don't believe their 'exception list' is a 'white list' or even 'essentially the same' as a 'white list' I can certainly understand the negativity in their direction ... They totally dropped one of the largest (if not the single largest) webmaster forums on the planet from their 'important list' and I think someday their going to look back and realize we dropped them right back...
</OffTopic>

[frontpage]

@ TheMadScientist

Good points. Google needs to remember that internet users are very fickle. Just ask the other search engine giants before Google that no longer exist.

[almighty monkey]

I think the claim that Google results are natural based on algorithms is pretty much debunked.

Google has stated that they hand manipulate results and that includes placing their own links at the top of SERPS.

This will be further weight to the government charges of a monopoly that is abusing its position.

Thing is, they can be telling the truth with this and still have an unfair advantage.

If the guys who wrote the algo were just down the hall, you can bet I'd be the top of every SERP I wanted as well. How on earth do you compete with that? They don't need to bother building an unfair advantage to the algo. The fact that they know it is advantage enough.

If there's one thing that is blatently off, its when Google run adwords campaigns for their own services. Even if they don't manipulate it, its not like it costs them anything. How are you supposed to compete with a campaign that has infinate money/