Where has it all gone wrong? Are big brands over-represented on the SERPs?

[Goforit]

What is wrong with Google?

A personal search, brings up just the same big brand names or the same reviews companies. I just want to be able to buy a product from a small company at a great price, like you used to be able to and they give good service.

I don't want a big brand that don't care after they have sold you it, at a sky high price. Also I don't want a million words to read about the product, I know what I want.

The same rubbish written in a slightly different way over and over again, in the top million pages.

This is not a search engine, it is so difficult to find something nowadays.

Yet Bing, nice and easy does it, type it in and a huge amount of sites, guess what? They are exactly what I want! Yipppppeeeeeeeee. 2 mins and that's it, ordered done and dusted, next day received it.

Why is Google making it so difficult to find anything?

Crazy Crazy World!

[tangor]

Ultimately, it's a market. And like all markets when there are so many sellers the customers can't see the forest, some of those sellers (websites) just won't be visible... only the largest will come through the noise.

And let's all be fair, there's so damn much noise on the web these days that it is amazing that sites are even listed, 100 pages down!

AFter all, there are only so many keywords out there (the extent of any language) and some will be more popular than others. If keywords are the sole property then serps are usuless, thus other ranking factors come in play... We are where we are for that reason and...

The OTHER factor:

Little fish are fodder for big fish, and big fish get bigger for that reason. Additionally, again this is ordinary marketplace, some fish get so big they can't be ignored and will dominate any given marketspace (space, not place!) The noise at the top is so much less than at the middle or the bottom.

Does that mean give up and quit? Not at all, just know that moving from page two to page one is an order of magnitude that must be addressed by whatever means/tools/advertising that is available.

There is an old adage that still holds true: You have to have money to make (a lot) of money. Mom and Pop websites have exactly the same ability of success as Aunt Martha's Thrift Shop on 4th Street, Podunk, Statetonia: A LOCAL market.

Google is international and all too often Mom and Pop's just don't rise to the top.

That's the commerce side.

Info side: Same thing, just no money changes hands. :)

[aristotle]

The great thing about the web is that it gives you an opportunity to reach a large audience at little expense. It's amazing to me how liitle it costs to register a domain name and obtain hosting.

Ten years ago Google's search results usually showed the highest quailty and most relevant articles at or near the top, regardless of whether they were on a big site or a small site. But then Google began getting frustrated because their attempts to fight spam kept failing. They finally decided that the only sure way to push spam down is to push big brands and big organizations to the top. But their early efforts to do this weren't very successful, and they eventually realized that they couldn't do it except by giving giving less weight to relevance and quality. So that's "where it all went wrong". Relevance and quality had to be sacrificed in order to get big brands and big organizations to the top of the results.

It's still possible for one person's independent work to rank above the pages of big brands and big organizations, but it's considerably harder now than it was ten years ago.

[EditorialGuy]

It's still possible for one person's independent work to rank above the pages of big brands and big organizations, but it's considerably harder now than it was ten years ago.

It isn't just a matter of competing against big brands and big organizations. There are a lot more sites of all sizes competing for the top slots, so the odds against ranking 1, 2, 3, or 50 in the SERPs would be a lot worse than they were 10 years ago even if Amazon, Home Depot, TripAdvisor, etc. didn't exist.

[aristotle]

EditorialGuy
Your comment fails to take account of the biggest reason why it's harder now, namely that Google's algorithm gives much less weight to relevance and quality now than it did ten years ago.

[rish3]

There are a lot more sites of all sizes competing for the top slots

In most of my e-commerce niches, there are actually fewer sites competing now, as compared to 10 years ago. Most of the smaller players folded.

What's different?

1) The skew of brand power, where, for example...Amazon will get a top 4 listing with a "brass widget" even when the search is for a "silver widget".

2) More manufacturer sites showing in the top 10, even when they don't sell direct to the public, and the query is clearly "buy oriented".

Fighting the brands isn't new for us, but G used to do a better job with relevance. I don't mind ranking below Amazon when the item listed is actually what the customer is searching for, and it's at a reasonable price.

I also don't mind ranking below a manufacturer for general terms like "widget", but if they don't sell direct, they shouldn't rank for "cheap widget" or "discount widget".

In short, the complaint isn't that "brands rank well", it's more like "brand seems to be over-weighted in the algorithm" in many cases.

[EditorialGuy]

Your comment fails to take account of the biggest reason why it's harder now, namely that Google's algorithm gives much less weight to relevance and quality now than it did ten years ago.

That's debatable, but in any case, one source (Internetlivestats dot com) estimates that the number of sites on the Web grew by 16 times from 2003 to 2013, so it's undeniable that there's a lot more competition than there once was. No matter how good Google's algorithm is or isn't, your odds of being at the top of the SERPs are far worse than they were in the happy days of yore.

[aristotle]

EditorialGuy
You appear to be missing the main point of this discussion, namely why did Google intentionally bias their search algorithm to favor big brands and big organizations.

[webcentric]

@aristotle Not that I agree with ED per se but I think you'll find that he doesn't agree with the premise e.g. intentional bias.

Not sure I do either unless generating profits is a bias.

[unity100]

That's debatable, but in any case, one source (Internetlivestats dot com) estimates that the number of sites on the Web grew by 16 times from 2003 to 2013

that's not a good statistic to evaluate search competition.

that is the period in which blogging has exploded. in that period until now, wordpress for example, become widespread and advanced so much that now you can just two click install a wordpress proper in a random cpanel web host. incidentally wordpress constitutes ~23% of all websites right now.

[w3techs.com...]

so, few of these are there to compete in search. you wont have competition from a random personal blog or 5 page website which was 2 click-installed by random guy somewhere to write about a few mumblings. these installs tend to not have the most basic meta tags leave aside seo plugins, so they are totally unlikely to rank.

that, and even for the better-built blogs and websites as such, there was and there is no possibility of competing for top keywords in first half of the first page of results.

that location is totally determined by google's algorithm changes.

[wheel]

Your comment fails to take account of the biggest reason why it's harder now, namely that Google's algorithm gives much less weight to relevance and quality now than it did ten years ago.

No it doesn't. My opinion is that exactly the opposite is the case.

It's pretty simple. Google figured out how to actually measure relevant and quality backlinks in their ranking. And they're doing a reasonably good job of it.

That's why all the big brands are ranking - they have high quality and relevant backlinks.

And it's why the small players, particularly the SEO-oriented, have disappeared. Because the quality and relevant backlinks they've been acquiring actually haven't been either.

You can argue that this is unintended - I don't think it is. You can argue it's bad, and I still don't think that's necessarily true. Small business owners complain like crazy, but when was the last time a consumer or the media gave a toot about Google's 'failed' rankings? The general public is perfectly fine with the current state of Google.

[aristotle]

Here is a statement that Google's Eric Schmidt made in 2008:

"Brands are the solution, not the problem," Mr. Schmidt said. "Brands are how you sort out the cesspool."[adage.com ]

That was when Google began to test various modifications and adjustments to their algorithm looking for ways to boost the rankings of big brands. As I mentioned earlier, they eventually had to reduce the weighting of quality and relevance in order to do so.

[n0tSEO]

@wheel - So are you saying that having brands outrank small businesses was always intended and nothing has gone wrong (to use this thread's main question), but it's just the way it's meant to be?

If I think of Google as a more elitist search engine instead of a general purpose one, I'll have to agree with you.

[EditorialGuy]

"Brands are the solution, not the problem," Mr. Schmidt said. "Brands are how you sort out the cesspool."

Yep, and what makes them brands? Often, it's innovation. (Amazon and Zappo's come to mind.) Sometimes, it's infrastructure and service. (If I buy a widget from Home Depot and it doesn't fit the whatsit in my kitchen, I can drive to a nearby store and return it within 90 days, no questions asked.) On the Web, content may also play a role. (Amazon's product reviews are the Yelp or TripAdvisor of e-commerce.) And, of course, success breeds more success through linking, positive user metrics, etc.

It's wishful thinking to assume that Google is biased toward brands, and that everything will be different if Google suddenly sees the light. In the real world, people are biased toward brands--often because of their own experiences--and how they interact with brands influences how those brands rank in search.

[aristotle]

It shouldn't take much thought to realize that Google can adjust their algorithm to create many different kinds of bias. In fact it would be difficult to create an algorithm that doesn't have any biases.

Google wanted big brands and big organizations to rank higher, and they expended a lot of effort, testing various changes and adjustments, toward that goal. Whether it made the search results better or worse is a topic for another thread. But it's very naive for anyone to think that there aren't any biases in Google's search algorithm.

[goodroi]

@EditorialGuy Are you claiming that Google has no bias towards big brands? That no part of their algorithm intentionally or unintentionally rewards big brands?

You seem to contradict yourself. You mention that brands generate "linking, positive user metrics, etc." and I think we can agree that Google rewards sites that do that. So either intentionally or unintentionally Google's current algo does seem to reward big brands. Looking at your own words it seems that it is not wishful thinking but very reasonable to assume that Google is biased towards brands.

I can see how people might debate how extreme the brand bias is or if it is an intentional vs unintentional bias in the current algo.

[netmeg]

Google is biased towards brands - yes.

All brands are ginormous, all-encompassing entities with virtually unlimited funds to create their brand - no.

A small, local business or website can't develop a successful brand without Google - hell no.

I don't really see anything happening in Google that doesn't happen in brick and mortar. A Wal-Mart, a Meijer or a Kroger moves into the neighborhood, and suddenly the smaller local guy has to scramble to keep up. Some people will stick with the local guy because they like his store, or they want to support a local biz or because they hate Wal-Mart and Kroger on general principles.

But usually *most* people end up going to the big stores, for a variety of reasons - it's cheaper, it's closer, it's more convenient, there's easy parking, they can consolidate trips to several stores into trips to one, whatever. First it was the malls, and then it was the megastores.

I don't particularly like it, but it's nothing new, nor should it be unexpected. Big fish are *always* gonna slurp up little fish, and us little fish have to figure out how to be agile enough to keep from being swallowed. Some will, some won't. That's gonna be true with or without Google. Google may speed up the process, but if it wasn't Google, it'd be some other search engine.

[EditorialGuy]

@EditorialGuy Are you claiming that Google has no bias towards big brands? That no part of their algorithm intentionally or unintentionally rewards big brands?

The term "bias" is a loaded word. The conspiracy theorists here would say (and have said) that Google is "biased" in favor of brands because this guy sits on that corporate board, corporate money types are in cahoots with other corporate money types, and so on.

What I'm simply saying is that Google's algorithm reflects users' wishes, preferences, and behavior. If millions of users are responding positively to Amazon, Home Depot, Macy's, Wikipedia, TripAdvisor, etc., that's going to have an influence on search rankings. And if Amazon ranks number one for a large number of e-commerce queries, it isn't because Larry Page and Sergey Brin are having a bromance à trois with Jeff Bezos. It's because search results are meant to be descriptive, not prescriptive.

[rish3]

The conspiracy theorists here would say (and have said) that Google is "biased" in favor of brands

No conspiracy required.

"Brands are the solution, not the problem...Brands are how you sort out the cesspool." - Eric Schmidt

[unity100]

Yep, and what makes them brands? Often, it's innovation.

thats a silly fairy-tale 'free market' belief.

there had been hundreds of tech companies which had great innovations, but their innovations were not able to do anything until they were bought by big brands, after which those innovations were presented in front of users as the big brand's innovation.

if 'innovation' did anything, there wouldnt be one google, one microsoft, one amazon - one or two of each and every major service. leaving aside the fact that what you call 'innovation' is something that eludes microsoft since ~10+ years.

......

billions of dollars make a brand.

an investment group basically bought and consolidated all the independent datacenters first into ev1servers and theplanet, then merged these two in softlayer entity, and then sold them to ibm.

until softlayer was sold to ibm, one major investment group basically dominated datacenter market in usa. without having any 'innovation', without having any experience, without knowing zit about the field. they just bought everyone out.

money talks.

back in early days, google was an upstart shunned by many megacorps. microsoft was everyone's darling. everyone did business with microsoft, google only had the search - everyone thought microsoft would eventually find a way to beat them - and microsoft did try - in a bazillion logical or illogical ways. all of which failed.

at this time google went with adwords, and exploded with adsense which basically spread google's advertising reach to billions. small webmasters, websites had made google. especially affiliate marketers, who were the darling of google in those early days. they basically poured billions of dollars into adsense. they made good profit too but thats beside the point.

google made heaps of money and gained the clout it has today at that period - financially, market-wise and respect-wise.

and suddenly major entities started advertising in adwords, or using adsense. when it was seen that google is not going away but growing, big businesses started getting cozy with google. shares were bought, traded, investments were made.

and suddenly "affiliates are an unnecessary middlemen" happened. whooops - in the blink of an eye google killed affiliate businesses, because they were not necessary anymore. now they had big brands to do business with. they were milking small businesses in adwords, and courting big businesses.

the adsense bidding war which google so fervently created and pushed through adwords tools and changes succeeded in driving all small businesses out at that period. now, the exposure belonged to those who could pay $4-5, and even $9 in some sectors like web hosting. from late 2003-2004 to mid 2005 that phase was over, if i remember right - even if bids did not reach whopass levels like $9 yet.

back in 2011, not only google was a huge player on internet, but it also had a lot of investments and dealings with major megacorps.

small websites were not necessary anymore. they could dot the 4th, 10th etc pages of search result all they wanted. in comes panda, schmid utters his obnoxious words about "brands sorting the cesspool" and fast forward to today where i still have to go to stackoverflow, stackoverflow, and stackoverflow my extremely specific, exact match keyword developer searches.

its a damn sql query. or htaccess rules. you CAN find some random 1 page result from 5 page 'thin' blog and give it to me so i can be happy.

but no, 'brands'.

well, there must be a reason why bing and yahoo are gaining market share, isnt it.

in the end, schmidt is ex oracle ceo. look at how oracle did and what oracle did, and you can guess the fate of google, leave aside understand their current actions. it even took so long to come to this point actually.

[wheel]

"Brands are the solution, not the problem...Brands are how you sort out the cesspool." - Eric Schmidt

And the cesspool is a big part of this. That's 90% of the websites out there, and perhaps 99% of the small business ecommerce websites. They're the ones trying to get rich by getting on page one and frankly, previously polluting the serps. Digitalpoint, warrior forum, every tom dick and harry buying .edu's, then directories then blog posts then the hoth.

Crying innovation or foul play is false and misdirection. We - small business owners - caused this. Everyone with an affiliate link or a small business decided they should be on page 1 and get rich....because they could. None of the results that used to rank were doing so because of innovation or any other differentiating factor. They were ranking because they bought into the latest link building scheme.

How many of you/us are actually providing something innovative? Almost none. How many of you should outrank the amazon when someone's looking to buy something? Almost none. Google's provided an alternative to the local shops - local SEO. Have fun. If you're national or international, then you should rightly be competing with the brands, and Google's serps are just fine.

And how many websites - including of the people here -are actually providing 'new' information? How many hired writers to write their content? How many hired someone to do their marketing via some link building scheme? Frankly, that's most of the small biz websites out there, they got to the top through laziness and luck. No more. I'm personally an expert in my industry, and my website writing reflects this - I never got to hire a writer to rehash topics. I had to take the time to write all that content myself, personally. My competitors took the easy route - and where are they now? Complaining about Google I guess (actually, they're calling me for a piece of my business - Got a call tomorrow from one).

I also recall repeated complaints about the serps previously, centered around 'spam' - search position above mine. Complaining about someone ranking for no other reason than they didn't like the small biz beating them. Well, google didn't like it either it turns out.

I remember the old days when I owned spot 1 and 2 for every term. I also remember all my competitors - some of them real dirtbags - getting ranked. Now they're all gone; every single one except me. And I'm holding on by tooth and nail in amongst all the multi-billion dollar international brand names in my industry. Happy? No. But frankly, the serps are where they should be, they got rid of the crud, and the only reason i still rank at all today is because I put a lot more effort in back when everyone else took the easy route. I put in my time when everyone got easy rankings, now I'm getting my payback and they're doing something else for a living.

The serps are exactly where they should be.

[goodroi]

Let's make sure we steer away from making editorial comments about brands and keep this conversation focused on the SEO aspect :)

[rish3]

We - small business owners - caused this.

I get where you are coming from.

However, "big brands" also participated in polluting the SERPS. And, when they get caught, Google usually offers a limited duration, limited pain punishment. That's well documented, lots of history to look at.

That is just one of many disparities. When a smaller business, that can't get a private audience with Google is slammed...there's no appeal process or limited duration punishment.

I remember the old days when I owned spot 1 and 2 for every term. I also remember all my competitors - some of them real dirtbags - getting ranked. Now they're all gone; every single one except me

For our e-com sites, we're in a similar spot. Us plus 2 or 3 competitors, in what used to be a field of 10+. But, none of us are doing as well as we used to. We're mostly sitting under brands with pages offering kinda/sorta similar stuff that is less relevant. Again, something like showing a "brass widget" when the user asked for "silver widgets".

What I suspect this is doing is causing more PPC clicks. Thus, perhaps, no incentive to fix anything, at least in the short term. Crap organics == More ad clicks.

[unity100]

Crying innovation or foul play is false and misdirection. We - small business owners - caused this. Everyone with an affiliate link or a small business decided they should be on page 1 and get rich....because they could.

thats not right.

every single megacorporation also did everything they could get away with, in order to grab more profits. this even includes violating laws and regulations in many cases. they even paid billions of dollars in fines - but its ok as long as you are making much more profit for every cent fine you paid.

leave aside cheating like big corporations - why should small businesses not even try to use the full extent of the means available to them?

there was nothing wrong with linking, backlinks and whatnot - it was the system of search, and even google was teaching everyone how to get backlinks and do it in which fashion.

so people bought paid backlinks. so what? how is that different from buying an ad? its also a backlink. what difference 4 links in a blogroll have from 4 ads on a page, except maybe the anchor text. google was not capable of discriminating in between them?

but ads are ok, since they were and still are fundamental means of financing the internet. so they never became 'bad', even today it is consensus (even on the side of google) to allow up to 3 ads as 'normal' on a page. shoving 3 loaded javascript on people's face which recall multiple servers and load multiple javascript to check and render one piece of image.

but its ok since it is needed. so all the concepts of speed, page size, snappiness are put on a back burner for these - if speed was a major factor, half of internet would be broken due to not being able to push ads.

no need to say that google has ads as a major business of theirs.

suddenly backlinks were decided to be 'spammy' if they were of 'low quality' sites and were many.

that killed A LOT of software companies - from web design to web app companies. overnight suddenly even the very popular, mega open source software outfits fell from search.

go now, try searching 'forum script'. find phpbb.com in the results - even the first mention of the word phpbb in anywhere in the serp comes in the 13th result.

a few much smaller projects are coming up higher, and totally irrelevant forum threads etc are coming up.

all because of the phpbb credit links in bazillions of forums that the phpbb project enabled.

now, according to the hilarious 'standards' google has brought to search, a major software like phpbb, which enabled and still enables a whopass amount of communities on internet, cannot even appear in top 10 of the serps for a specific, exactly relevant query.

'brands' have won, search has lost.

How many of you/us are actually providing something innovative? Almost none.

where are the ones who DID provide spectacular innovation today?

go check google's, microsoft's, amazon's, ebay's acquisition lists. you will find almost all of them there. their innovations are there too, either utilized or not utilized, either waiting the time they will be pulled out to be used or bought to prevent them from becoming risks in future.

why didnt these innovators with a lot of contacts, connections, - some even exposure - even disruptive devices did not become brands or get to top of serps? then and today?

Complaining about someone ranking for no other reason than they didn't like the small biz beating them. Well, google didn't like it either it turns out.

of course. especially when you have investments and shares in those other companies.

and that alone totally invalidates any self-blame like 'innovation'.

your innovation is their profit loss. therefore you are a problem which should never happen.

in conclusion it is quite silly to trust any private company to control and handle critical infrastructure or anything that is necessary to the conduct of life and business. google was basically the search monopoly, it still is the search monopoly. 60%~ is market share IS monopoly. you may not control what every single person does or have their business, but you do control the trends, the finances, the economics of the game. and in the past decade we saw how did this work out.

its not easy to blame them for anything since this is the rule of the game - not utilizing one of their services to maximize their profits and investments in other services would be foolish.

they couldnt act like a charity to provide an even playground for small businesses and big businesses.

long story short, this was bound to happen regardless of how anyone/everyone behaved

things like these make me think internet should be a total public utility. its basically like roads now - bread and butter.

[unity100]

i forgot to touch an important point, sorry:

And how many websites - including of the people here -are actually providing 'new' information? How many hired writers to write their content? How many hired someone to do their marketing via some link building scheme? Frankly, that's most of the small biz websites out there, they got to the top through laziness and luck. No more. I'm personally an expert in my industry, and my website writing reflects this - I never got to hire a writer to rehash topics.

i provided original information, wrote what little article i wrote myself, and im pretty knowledgeable not only in my field but also a LOT of other fields due to an aptitude for research in certain topics. since early 2000s.

and now today i regret not engaging in backlink building back in those days when everyone was doing it, and playing it proper and 'natural'. i didnt benefit from doing as such at all.

that attitude neither did provide any benefit AFTER panda/penguin, mind that.

i still provide very unique information, some of the articles i write can hold even extremely-bouncy social media visitors like those from stumbleupon on my article (they are too itchy to click the stumble button and go away from the page out of habit), various of my articles recieved close to 50,000 stumbles, shares, likes in total, some surpassed 100,000, domains are clean, doesnt have any scammy backlinks etc. one of my articles even randomly made google news (totally unintended) for something that there was little news in the english media...

and yet still no search traffic to THOSE articles. so, social was a weighted ranking factor? say what?

thank god there are sources like stumbleupon which bring interested readers. its not like there is any profit, for this kind of interest/hobby/expertise visitors do not finance anything. (with 'this kind' i dont mean niche interest, but social media traffic). but at least what you write reaches out to people and does not go to waste.

[toidi]

Are big brands over-represented on the SERPs?

from a non engineer/developer perspective, the time frame when the big brands took over was a time when the serps were falling apart from the massive spamming going on. Relying on the big brands made it look like the algo wasn't broken.

[Nutterum]

If you generate that much interest @unity100, why do you need organic traffic? And more to the note, do you think the organic traffic will be more willing to buy/click/whatever to your services or products?

To me what you just said was "I am entitled to traffic!" when in actuality, no one is. And more to the point of brands. If you are getting such high amount of social traffic to your articles, why not make a brand. Why not promote your website as a brand, make mobile app just like a brand etc. etc.

If you so much believe that you write stuff more specific and better than the top dogs, why not distinguish your self from them by becoming your own entity?

See Google does not care! Their algorithm is made in a way as to detect when something is invested in to, be it money, professionalism, unique selling point, you name it and the algo puts that thing on TOP, because it assumes(correctly I might add) that the information or product inside that full of effort and investment website will satisfy the needs of the many.

No conspiracy theories. No bias. No back of the room big buck-corporate-share talks. Its all about math. And the math say and always pointed that in order to be be big, no matter the space you occupy, you have to act big or win the lottery. And sure as hell Panda and Penguin make sure you don't win the lottery!

[Rlilly]

The writing was on the wall in 2011 when Google started giving brands the first 8 listings (domain crowding) on a page including 6 sitelinks for the first 2 listings.

[rish3]

the time frame when the big brands took over was a time when the serps were falling apart from the massive spamming going on. Relying on the big brands made it look like the algo wasn't broken.

That seems about right.

Google can fairly easily correlate websites to either "publicly traded company on a major exchange" or "big investment from a known VC firm". Search google for "Walmart" or "Andreessen Horowitz", and you'll see the "People also search for" widget in the right hand sidebar. That, and just a bit more is enough to determine if a website is a "brand".

If your algo can't sort out the quality results using just signals, it's easy enough to overweight brands as a band-aid.

Edit: If you want an example of this, search Google U.S for "Women's Stretch T-Shirt". You'll find a listing from the Berkshire Hathaway company store around #10 or so. The store is really meant as an "internal / employee use" website...it's not a real contender in apparal ecommerce. But, the brand-boost lets it rank well with little effort.

[edited by: rish3 at 3:00 pm (utc) on Feb 18, 2015]

[Nutterum]

OK. Let us assume that the people who believe the brand overweight is there as a manual plug are correct. Why do you see them dominating competitive SERPS as well? Bing and Yahoo are dominated by big brands as well. Its just that the SEPRS below position 4 are different (in good chunk of the cases) than Google. Did their search engines fell in the same loophole? Even Yandex is dominated by brands, russian brands but still brands.

[rish3]

Why do you see them dominating competitive SERPS as well?

The nuance here is that brands can rank on Google, regardless of quality or relevance.

Brands dominated before the Vince update, and other updates that have provided an artificial boost.

It's just that now, they can outrank specialty stores for a "specific widget" with an auto-generated listing for a "general purpose widget".