There are fundamentally two categories of websites -- those that provide information (think Wikipedia, the BBC and Frommers) and those that sell something (think Amazon and eBay).
Participants in this forum work on both types of sites, many ecommerce sites also provide information, and many information-providing sites generate revenue by selling ads or through an affiliation with someone who sells things. What all of the participants in this forum have in common is that over the years we have all spent uncounted hours trying to figure out how to get more traffic from Google by using various techniques -- most of which fall into the broad category of activity referred to as "SEO."
In turn, Google hasn't been pleased with many of those efforts -- primarily because SEO is the primary expertise of spammers, and the exploitation of various weaknesses in Google's system have allowed various forms of spam and garbage to appear on the first page or two of the SERPs -- embarrassing Google and irritating them, since money being spent on SEO is money that is being spent on Adwords.
Although Matt Cutts only talks about fighting "spam" Google efforts in that regard have had the effect of making all forms of SEO more costly and more risky.
The end result has been to reduce the value of virtually all types of promotion and marketing on the web -- virtually any method used by small sites to try to get traffic other than purchasing Adwords has become more risky, because today's "white hat" SEO may become tomorrow's "unnatural manipulation" of the SERPs.
Not coincidentally, the ongoing battle against spam has made it harder to get traffic except by paying Google (Adwords and listings in Google Shopping, etc.)
But, I don't think this is the primary focus of Google's efforts, nor do I think the current situation is stable.
Google probably knows, or soon will realize, that it has gone too far in pushing nearly all small sites down off the first page. It's not just the spammers that have been pushed down, it's all sorts of small sites that are mistaken for spammers or are sending off signals of attempting to promote their site through SEO. But, these firms were using SEO that was once considered acceptable, or at least hard to detect, but those same forms of SEO are now considered to be manipulative (or at least Google is noticing them more).
This has led to an extreme imbalance in favor of large, branded sites and against small sites that aren't household names, and can't afford to advertise on TV. Interestingly, the problem is affecting both information sites:
...in the travel and tourism sector ...over a period of several days I was running hundreds of searches that involved most countries and major cities, along with their most popular attractions, transport networks, restaurant guides etc etc.
The overwhelming result from all that searching was ...the same group of authority (aka big brand) sites in almost every single search.
and Ecommerce sites:
The SERP's are also full of eBay & Amazon listings. ...How can it be in Google's interest to list eBay & Amazon so much? Keep doing that & even the lazy "searchers" will just type the URL in their address bar cutting Google from the loop.
These quotes confirm observations I've made in my own niche -- both types of small sites are suffering. These sites don't have the marketing heft or budgets to run TV ads or otherwise get traffic, and they can't make money without a strong flow of traffic from the search engines -- and Google controls the vast majority of that traffic (70-90% depending on the country and niche).
But I don't think this imbalance can last much longer, because it fundamentally undermines the value provided by a search engine in the first place -- helping users find things they would have difficulty finding on their own.
Times are tough, but I believe this too will pass -- the current situation is NOT a stable equilibrium.
For its own long term interests Google HAS to finally figure out how to identify and list good small/specialist sites on the first page of its results. The problem is that it has never figured out how to find the true authorities/specialists in small niches -- it has always gotten the signals of authority/quality confused with the signals of successful SEO efforts, which in turn has led to confusion about spam/garbage sites that are very good at SEO, or do it on an enormous highly efficient scale.
The end result is that over the past few years Google keeps doubling down on its attempts to stamp out spam, and any form of SEO which it feels is too manipulative -- and the "safe harbour" of acceptable SEO has gotten smaller and smaller.
The end result is that it has become extremely difficult for small sites to survive financially.
Whether or not SEO will survive as a viable form of marketing is an open question. Conceivably, Google will create so much risk that SEO will become too risky to be cost effective.
But one way or another, I believe the environment has to eventually change to allow smaller specialist sites to become profitable again. The current situation, with large brands dominating the first page of so many searches, while smaller more specialized sites are pushed out of view is not fundamentally stable. It detracts too much from the essence of what Google does as a search engine -- helping users find obscure sites they don't already know about, and couldn't find on their own, without using a search engine.
