Authority sites ranking for thousands of site search pages

[Andem]

I've been wanting to post about this for a while, but I figured it would resolve itself after a couple of months but that hasn't happened yet. This is different from tag pages, continue reading.

There are several high-quality sites which have taken advantage of their status as an authority in order to link to search result pages from other parts of their site. These pages eventually rank and thanks to Google's abandonment of the "one or two pages from the same site" on the SERPS, I'm running into page 1, 2, 3, 4 and above which contain 90%+ of internal search result pages from authority sites.

I really got annoyed today when I was looking for something that only we offer (a 100% free, non-ad-supported service and trademark) and it appears that one site has created search result pages from keyword lists and now ranks 1st page, often above the real site for countless terms.

The general layout of virtually all of these pages are search results, 2-3 AdSense for Search custom ads as the first two "results" and then mildly rated pages to completely unrelated pages (many are registered trademarks). URLS look like:

site.com/s/this-and-that-for-the-other

The problem is that these sites are using their authority status to rank for stuff that they don't have, nor do they offer. The stuff they rank for really looks like they are generated from database compiled from Alexa or AdWords data. One site has over 50 million search result pages listed while others have between 10-40 million. Ridiculous.

This has been going on for months and it isn't fair to anybody. I've used the G Search feedback form several times hoping to raise some flags because it is frustrating as both a user and an original content producer.

[tedster]

As a user I agree that it's frustrating to click on a Google result and find a page that tells me the site DOESN'T have that item. I don't see this ranking as a competitive advantage however. After it happens to me a couple times, I never click on that site's result again. And of course, even the first time, I still need to go back to the SERP and make another choice.

[Zivush]

I have seen these keywords links in one of my toughest competitor site - [the well known about ..etc].
When clicked you are taken to the site's search page, which is actually "Results by Google Custom Search".
To my best knowledge, unless a redirect trick is implemented, this is a kind of feature given to premium Adsense publishers because the url looks real - example.com/lr/popular-keyword/1288309/4/

I haven't seen these type of pages on organic results, but if they are (as expressed here), it is an unfair benefit given to this site no matter how authoritative it is.

[austtr]

These duplicated-site results have started since the Panda and Penguin implementation. To see comments from Singh/Cutts that they are happy with the reults since those changes is beyond the belief of this particular poster.

This is corporate twaddle at its worst... the type of patronising nonsense aimed at an audience that won't know the difference. I look forward to the day when folks at Google will discover their lost testicular fortitude and discuss what is obvious to blind Freddy.

[indyank]

I have made a post on this here more than a year ago. This is a very common practice with very high authority sites. The information architecture is laid out exactly as stated by Andem and these search result pages (which are infact carefully created from several related keyword phrases) are the prime entry pages for their search engine traffic. However, this is an age old practice and not something which is just a few months.

Andem, another thing you would see is these auto-generated search result pages (generated from a bunch of related key phrases) with url structure as you mentioned, would interlink to other similar search result pages as "Related searches" using those same bunch of related key phrases. They also somehow manage to automatically generate unique titles and meta descriptions for these groups of pages and these titles and meta descriptions differ only by one or two words in most cases.

If they have say 15 to 20 products for the primary keyword "widget" they would list these products in random sort orders for each of those related search result pages with titles like "widget for ...", "....of widgets" etc.

They continue to flourish after all these changes by google in the last one year and their traffic only seem to have sky rocketed.

austtr, I agree with you that this kind of site architecture and navigation is very common in some niches and they have been given a real traffic boost after Panda.These are professionally designed high authority sites and that seem to help rank their auto generated search result pages.

[deadsea]

Back before Panda and Penguin, I implemented this kind of thing for some authority sites. A couple of the sites that I implemented it for were hit by Panda because of it. They took it down and recovered from Panda eventually.

One authority site that I worked on still has it up. It is the most interesting case because they try hard to only target relevant keywords and have put a lot of effort into building the pages to look less like search results and more like product pages. Its great for them because it creates landing pages for alternate ways people search for their products.

If you are going to implement this, here is my advice:

1. Google doesn't like it, and this can cause your site to lose its authority and rankings.

2. Don't use the word "search" anywhere on the page (other than the search box where you use it on all your pages). Its a red flag to Google if especially if the word "search" or "results" appears in the title.

3. Be very careful about what you can create pages for. Only create pages for relevant terms. Have your tech implement some confidence threshold and if it is below that, return a 404 instead of showing a off-topic page. Matt Cutts has publicly demonstrated that a site "deserves" a penalty because he was able to type a celebrity name into a search box (way off topic for that site) and get a crawlable page.

4. For a search term that is an exact match for the name of a product, 301 redirect to that product rather than create a duplicate page.

5. Make the page look less like search results. Often people are interested in just the very first result. Pull in additional information about that result. Make the page look like a product page in these cases. Pay attention to user engagement metrics for this the same way that you would on the main product page.


If you are going to report sites that are doing this, try to create a really off topic page. A current event might work well as you might be able to link to it and get it indexed quickly where it gets enough visibility to bring it to Google's attention. Link to that page from somewhere to get it indexed if you have to.

[Andem]

this is a kind of feature given to premium Adsense publishers because the url looks real - example.com/lr/popular-keyword/1288309/4/

These are not Google custom results. The pages simply have custom AdSense for Search ads.

They also somehow manage to automatically generate unique titles and meta descriptions

The meta descriptions are sometimes selected from a phrase which appears on the first search result page.

If you are going to report sites that are doing this, try to create a really off topic page. A current event might work well as you might be able to link to it and get it indexed quickly where it gets enough visibility to bring it to Google's attention. Link to that page from somewhere to get it indexed if you have to.

With perhaps tens-to-hundreds of thousands of unique visits from the SERPs per DAY, I'm sure Google has already taken notice of these results and done nothing about it.

I wanted to add: One of the sites I see the most also uses authorship markup for search results and gets an SERP listing with their profile headshot.

[netmeg]

In my experience, Google may do something about it and may not. If the site has enough authority, Google may figure that search results pages from an authority site are still better than crap pages from a crap site. And if the percentage of good pages is significantly higher than the percentage of crap pages, the company may still get a pass.

Plus, we all know it takes Google a while to get around to everyone. In some cases, it's been years and years.

As far as the people in charge of these sites, in most cases they are probably aware of what they are doing, but I have consulted with companies (even big companies) who had no idea that their search results didn't belong in Google's search results.

It's not a strategy I would recommend, for anyone. Pretty much the first thing I do on any site is remove/block the search results page. Mostly just because I hate seeing them in Google myself.

[Zivush]

These are not Google custom results. The pages simply have custom AdSense for Search ads.

Just checked again. You are right. What a way to create more pages (and drive clicks to ads).

[atlrus]

I have seen this with Walmart.
I would search for a product that would never sell at a place like Walmart, yet google ranks them first and the page consist of "Your search for Product did not match any bla-bla-bla..." And that was before google opened the duplicate floodgates.

[1script]

Back in 2006 I had a site banned from Google for 2 years for just that - making site search pages index-able and then a page with a list of the most popular keywords was linking directly to the site search pages. These then become the classic doorway pages that Google does not like. In fact, Matt Cutts had specifically spoken out against having those search pages indexable. Unfortunately for me, it was after my site was already banned.

Fast forward to 2012. I am habitually adding instructions to robots.txt to disallow search queries. However, most of my competitors (*how sites and clones) are ranking (and outranking me) with what looks like either a search result page or an empty content page that only has the keyword in the title and H1 and looks like they either plan on filling it out with content later or just simply creating hundreds of thousands of doorway pages that way. Sometimes you can see a page that says "No results available" and it still ranks!

And, yes, it was a very astute observation: all these sites have a custom AdSense code, not the regular one. I think I even have a "less sinister" explanation for why they are getting a pass. It looks to me that since the custom AdSense is less clearly marked as an ad and they have more leeway in styling it similar to the rest of the site, fewer people complain to Google and report those pages as spam.

It's either that or their AdSense account reps do indeed meddle on their behalf. And they don't even need to do much. They don't really need to alter any algos to make the doorway pages rank. All they need to do is to call the spam team and say:"these guys are OK". Most of what they rank for is very long tail and does not take much to rank for. This way everybody gets to keep their face and say that being a premium AdSense publisher does not affect your rankings. It's true, it doesn't. It just affects whether you'll be wiped off the face of Google for the serious infraction or not.

And, by the way, what a novel concept: an employee of a large corporation picks up her phone and dials a 5-digit extension of another employee or meets her at the water cooler. Millions of dollars of revenue can be affected by one little chat like that and it does not even qualify for insider trading, so can be done with complete impunity. Are we all so naive that we should believe this *never* happens?

[netmeg]

I think it's probably rare, at least on the Google end.

We seem to be talking about two different things here.

There are search results pages that actually have search results on them. Arguably, it's content - it may be thin, shallow, crap content, but it's still not a blank page.

And there are search results pages (or other dynamically generated taxonomy pages, for that matter) that are blank - either the product is out of stock, or the event has expired, or it's a 'placeholder' page waiting for future content. These pages have no content.

None of them should be in the index, of course, but personally I object more to the blank pages than the others. For my event taxonomy pages, we had to write code that would slap a NOINDEX on any page after the last event expired (and also remove any ads) But that's not usually built into a CMS; you have to be able to write something custom for it.

The Google CSE results pages *should* be an easy fix if Google didn't really want them in the SERPs; they provide the code, they can recognize the signature, so they should be able to deindex them on sight. I dunno why they don't do that. Of course the cynical answer is, they have ads on them (required by Google, by the way) but I can't imagine they get enough revenue off ads on blank search results pages to make it worth the loss in quality.

[indyank]

yes netmeg we are indeed talking about two different things if not more.Even the first one that you had mentioned are dynamically generated and do have the same content, say 10 products relevant for those searches, arranged in random orders.Each of those products will have a line of description (most times partially cut off) and other stats about them like the number of views, etc.

From whatever I have seen, these sites don't use the term "search" in their url structure and the structure is more like "site.com/s/this-and-that-for-the-other". They also don't use the word "search" in the title tag, though they would mention it just outsite the h1 tags but styled as h1 tags. For example, "<h1> Widget </h1> Search results"

You are right that google could easily detect these but the fact is most of these are high authority sites and so we are always going to find custom adsense codes on them.

[netmeg]

I am not sure what you mean by custom adsense codes. Are you talking about Google's Custom Search Engine (or AdSense for Search) product? Or are you talking about Premium AdSense Publishers, and some other search product?

[1script]

@netmeg and @indyank: Guys, we are definitely talking about premium AdSense publishers, *not* CSA. I've never seen CSA results in Google SERPs and I also believe it's an easy fix for them if any of those pages ever creeped in. Premium publishers are able to change many things in the ads and especially two of the most important: the exact size of the block and the fonts of the titles. Everyone has their eyes trained on 300x250 blocks with Verdana titles and it takes people 0.1 seconds to realize that this is an ad and, if they feel really annoyed, report that to Google. Premium publishers are able to blend in much easier and I have no doubt they get reported less.

Anyhow, getting back to the issues of search pages in Google's SERPs: indyank is right, they don't always have the word "search" in their URL or content - perhaps this is what lets them fly under the radar? They are still dynamically generated doorway pages that contain no information about the product/issue, only something that is "related", usually a list of links to other pages, and sometimes it is only so vaguely related that you're amazed it's ranking for the product name with just a title and H1. BTW, indyank, I should check if it's really H1 - you're right, it may just be styled as the largest header but not marked as H1.

I did, however, come across some of those ranking search pages that do indeed say "search for #*$!" on them. There can also be "tag" pages ranking etc. So I would call all these "doorway pages" which is what they are - they are only leading to other pages of the site. And that's in best case scenario because sometimes there's nothing to lead to, yet they still rank.

Bottom line: Google lets large sites get away with doorway pages - something that was a no-no since the dawn of Google.

[netmeg]

(There is no CSA product; there are Google CSEs, and AdSense for Search. Just so you know.)

[1script]

CSE (Custom Search Engine); CSA was a typo.

[superclown2]

This obession with 'brands' or 'authority sites' is a huge weakness of Google's which will lose them massive market share when there are credible alternatives which there will be, sooner than they think. Throughout history market leaders have ignored the tripwires, that everyone else can see clearly, before they fall over them which is why they rise and then, inevitably, sink into oblivion. Remember GEC, Ferranti, Fairey Aviation, Hoover, Altavista, ICI, BMC (etc etc ad infinitun) anyone?

[netmeg]

The obsession with brand or authority isn't just a Google thing, it's a people thing.

[spreporter]

yet obsession or fashion or whatever is temporary....

[Andem]

The obsession with brand or authority isn't just a Google thing, it's a people thing.

yet obsession or fashion or whatever is temporary....

I'm quite sure that the majority of average users stumbling on to a site's search results don't view that site as an authority, especially those who know that they are doorway pages. In fact, many of these sites are only viewed as an authority by Google.

[netmeg]

There are an awful lot of people who think of eHow and About.com as authority sites. Don't be too sure.

[superclown2]

The obsession with brand or authority isn't just a Google thing, it's a people thing.

Most people are sheep and sheep will go where they are led until another 'leader' offering better grazing comes along.

If Google continues to take the easy way out and continues to lead them to brands then that is where they will graze comfortably, unaware that there is no nutrition in what they are consuming. Eventually they will wither away.

If Google wants a long term business then they will have to get away from this easy fix, there are other intelligent, highly motivated people out there working on ways of taking their business from them and things change fast on the Internet.

Perhaps they could offer more than one (non monetized, and therefore hopefully impartial) algo choice? One with the current brand oriented algo and one concentrating on information sites would be a start. They would benefit from a higher number of searches as well as closing a path that a potential rival could use, and those of us who wish to could perhaps find out what these products or services are all about instead of facing a long list of better known companies that give us the same plain vanilla information about something they don't even produce or sell themselves.

Google is the market leader. It must show leadership if it wishes to survive and prosper long term.

[miozio]

The search is flooded by coupon and torrent sites that lead users to ads, spam and even malicious pages. These appear on the first page for Ourbrand.com search, sometimes even as high as number 6 result. They don't have our coupons and they don't seed our copyrighted films. It's just an UTTER SPAM! Those torrent sites like spamtorrent.com/ourbrand.com/ claim our film downloads for free with links such as "trusted download" where they mislead our clients and make them confused and possibly ripped off! I hate this! This started happening about a month ago.

[indyank]

Yes I meant premium adsense publishers...They have the luxury of customizing the look and feel of their ads....so they customize their ads to look like one of the results on their search results page.They also place them at the top of their search results!

One think I don't understand is when one part of the Google them (the search team) is convinced that certain ads which occupy a large portion of the real estate at the top of a page is spammy, why doesn't the google adsense team offer all their publishers the ability to define the size of the ad?

[Zivush]

why doesn't the Google Adsense team offer all their publishers the ability to define the size of the ad

That is The question.
I think they genuinely want ads look like ads for the the vast majority of publishers mainly to protect their advertisers from invalid clicks.
When it comes to brands and big sites, the picture is changed and other factors start to play.
[[OK, moved this part of the discussion to Adsense forum hoping that Adsensers will read it]]

[edited by: Zivush at 5:02 am (utc) on Aug 7, 2012]

[tedster]

Come on guys - we have do an Adsense forum. That's where this discussion belongs. This thread is about Google indexing Site Search results.

[indyank]

Tedster, sorry if it sounded like a pure adsense discussion. But the google search algorithms do have ads as a factor and the discussion was on search result pages of authority sites that also seem to have disguised (customized) ads as part of their product lists. Yet, neither above the fold disguised ads nor the fact that search result pages are allowed to be indexed and interlinked massively with other similar auto generated pages, seem to affect their ranking!

It was really difficult to keep the ad part of the discussion separate.

[spreporter]

"this easy fix" IMO this is the problem.
Probably it was a kind of easy fix to the algo in their effort to fight spam. I'm sure they know that there are out there hundreds of great websites with information dedicated to the subject and not the item a, b, c, of www.brandsite.com -but- it might be the case that they are satisfied with their profits that coming from the current results, at the moment, or, with the current link/anchor text algorithm they facing problems to discover the good "non authority" websites (or they devalue them) therefore they just through the baby with the water and you get served by the easy results you see today. Finally is not Google to blame but the black hat community, the blogosphere and the uncover of JCP spam that led to the current updates and situation.

[spreporter]

and yet, all might change with a new algo update, who knows....