The State of the Internet (2013) - Summarised perspectives.

[hitchhiker]

My personal observations, and summary of information taken from a number of forums, and the personal observations of 20 or so webmasters with a combined traffic of 10s of millions of visits per day.

I've tried (my best) to summarise what I've seen in relation to this, and not editoralise too much.

Disclaimer to newer folk: Take this for what it is, a set of observations - somewhat muddied by our personal experience and expectations. We do not work for Google, we only have part of the picture.

Summary

A fairly clear pattern has emerged: The fight against spam was HUGE. At some level Google failed to react to the crossfire generated, and collateral damage. It hit widely, but no one more than UGC (User Generated Content). In part made possible by the narrowed gathering of, and seeming reluctance to consider, wider-perspective webmaster feedback.

Why all this UGC?

We are normally an order of magnitude bigger than everyone else. Things hit us first and more noticeably. Our area is 'gray', both quality and otherwise. We fight SPAM at the site level and tend to see updates before others do (due to our reach and size). In some regards, we may signal what's to come for SMEs (small and medium enterprises).

UGC site owners are hurting (pretty much across the board).

Even StackOverflow seems to have been hit in 2013 (according to alexa*)

"Too many updates, they're being careless." - 500 a year according to Matt.

WebmasterWorld (alexa* again) declining since 2011 (it's safe, don't worry, we all love it here!)

Reported: Perhaps a general decline in search traffic via Google (now at #2 spot on Alexa - Facebook #1)

Quality doesn't seem to be much of a factor. Many authorities hit hard.

No clear examples (so far) of older UGC that survived.

Examples of things like 'blank' sites, double H1, '2005 black hat SEO' making it into the top 10 (a lot of this seems to be done at the bottom of the page) Read the thread, it's quite entertaining*

Bing showing very different results.

Panda provided a boost for a lot of us UGC. I saw my traffic rise through 2011-2012. (21.5 nov 12 corrected that.)

(Unsure) Custom software, or updating your look may have resulted in another inadvertant penalty (*cross-fire, sticking your neck out)

Non-UGC: Manual adjustments for keyphrases patching and semi-correcting the problem here and there.

"Hard to say where traffic drops are occuring, it's just 'everywhere'"

Links are at the centre of this storm. UGC Webmasters forced to No-Follow everything meant organic user preference from the wider audience was lost. Hurts everyone.

The google product forums have unintentionally become an abysmal way to keep us from getting answers. (Our experience as we ventured in there recently to hunt down some answers)

Brands may not be favoured, but at this level of chaos - they're more likely to survive it.

According to some people: Black hat has become non-viable. If that's true, it may have been done by over-reaching on too many signal patterns.

I've left a lot out (paid links), as I haven't personally been following it. This is just my own meandering personal perspective.

* "Cross-fire collateral damage." In a time when bullets are flying around you, keep your head down. In web terms, that would mean blending in with the crowd. "Use traditional software, don't do anything custom-made. No-Follow every link." That's not a great situation, that hurts everyone.

* Alexa works pretty well (in my experience) for large sites - those of us in the top 10,000

* Some entertainment: [webmasterworld.com...]

Fully personal opinion

I'm a white hat SEO, for 15 years. First website made in 1994. IMHO: Good content has never been hit so hard, and to this extent. Some may argue, the current state of affairs will result in the loss of smaller deserving businesses, those without the deep pockets to survive turmoil to this degree.

IMHO Focus on content, even now. Nothing else has a longer-term chance of survival.

I don't believe any of this is intentional on Google's part, I believe them when they say they don't favour brands. PPC would be a better place for brands in general (they are used to that and can afford it, everyone wins). Leave organic to a fair mix of SME and brand - then the ad space used above the SERPs would be acceptable.


Question to the community

---

Do we need, as a community, to establish better dialogue with Google? If so, a proper way to do that. To keep the signal to noise ratio down. Real Googlers participating, hundreds (not 2). I believe some inside the web-spam team might agree, please speak up.

---

[edited by: goodroi at 1:18 pm (utc) on Jun 12, 2013]
[edit reason] per author's request, added question to the community [/edit]

[chrisv1963]

Spam works, blackhat works, we all see it.

The only thing that no longer works after Panda and Penguin is good original content and being successful by following Google's webmasters guidelines. But who cares ... Google is "excited" because of all the "progress" made since February 2011. What used to be a great search engine is now a heaven for spammers and scrapers.

[aristotle]

At one time relevance, quality and usefulness were major factors in Google's ranking algorithm. But these factors were an obstacle to Google's goal of getting big brands and big organizations to the top of the search results. So in order to achieve its goal, Google had to greatly reduce the importance of these factors in the algorithm. In a nutshell that's why Google's current search results are so bad.

[dataguy]

I really think that the issue here is summed up by stating that Larry Page is just not very mature. The issues that we, as webmasters, are seeing at Google, won't be resolved until a more business-like approach is taken at Google. Sadly, this won't be fixed until Larry Page steps down.

And this isn't just about us webmasters. Google users in general are starting to figure this out. I know people who have switched to Lycos recently, so what does that tell you?

[hitchhiker]

If any of you read the OP - it would be clear there's a fairly wide field of data showing why and how the Webspam team have dropped the ball. Harping on about how 'Google is evil' is useless to us.

If you don't want real communication, fair enough - this is the place to make that clear. Getting stuff off your chest amounts to trolling us at this point.

SMEs and UGC are hit hardest.

There is little evidence that brands are selectively targeted, but it's clear with this much confusion that not much else will survive.

This thread is about whether or not we need to communicate with Google, how we can do it, and if it's worth doing. The last two pages are clear reasons why that's not going to be an easy task.

A lot of you need to now consider the 'BRANDING' route. As discussed here: [webmasterworld.com...]

If you can't go that route (forums / Q&A sites) then it's probably in your interest to help us garner support for getting REAL ANSWERS to our most serious questions.

This will almost certainly be the first and last coherent thread that you can do that on for a while.

[goodroi]

hitchhiker stop trying to get webmasters to work on being more profitable. I prefer when people waste their breath screaming that life isn't fair because money isn't falling out of the sky into their laps. It makes it so much easier on me to outrank them in the search results.

I wish they keep on whining about how much Google sucks. Those comments are not useless, they make me lots of money by distracting my potential competitors from doing real research with proper experiments and scientifically reverse engineering the serps to notice trends that would enable them to realize what is needed to run a successful online business.

[diberry]

They are being allowed to do a lot of this BECAUSE we haven't proactively pushed for any two-way communication.

Uh, actually we used to HAVE that kind of communication, many years ago, and Google walked away from it.

And if you think "Google bashing" is the reason they walked away, it's just not. Google is going to communicate with us exactly as much as serves their needs and their purpose. Google has very recently begun to improve their communication with us in some small ways, but only ways that serve them.

If you want Google to dialog with us, you need to convince them THEY will benefit from it. Believe me, if Google felt they were getting a lot out of a dialog with us, a few negative posters would NOT deter them.

[heisje]

reverse engineering the serps

really? in 2013? - well, lucky you . . . .

.

[seoskunk]

Ok lets leave aside the spying and profits issue for other threads, hitchhiker has clearly put allott of thought into this post and research.

I think the number one issue would be how do we "vote" for questions to be put to Google on Webmasterworld, can a poll be set up?

Historically answers came via Google Guy in this forum, perhaps that account can be resurrected and it doesn't have to Mr Cutts that answers questions, but a anonymous engineers from the plex,

I think this is worth a try I mean what's there to loose. Webmasters need some guidance to get back on track thats for sure.

Perhaps now is a time for hitchhiker to create a new thread "Questions for Google Guy" and let people post their questions.

[hitchhiker]

@goodroi: Those comments are not useless, they make me lots of money by distracting my potential competitors from doing real research

ROFL, man you're so spot on here.

@diberry: you need to convince them THEY will benefit from it

I've been trying. I have to believe they care about the state of the SERPS. So: Feedback from the community (we saw these things start a year ago). What the h*** else can we offer them? :)

@seoskunk: Perhaps now is a time for hitchhiker to create a new thread "Questions for Google Guy" and let people post their questions.

First off, thanks for the post. I'm waiting for the go ahead from WebmasterWorld; it's currently a gray area as the forum charter specifically states (paraphrase) 'this is not a place to attempt to contact Google directly'

I think dropping suggestions here *might* be ok though.. If you look back at page 4 there's these:


"We tend to see very little evidence of recoveries via the disavow tool, and it's existence somewhat calls into question the current implementation of 'PageRank'. Isn't this just a 'hack', a somewhat clumsy attempt to fix larger problems; one that needlessly distracts and confuses webmasters?"

I believe 'Negative SEO/toxic links/disavowing' should have been something you vehemently guarded against. (The concept of any link or relationship as being 'negative' rather than 'ignored') No external entity should have been able to affect our websites negatively. Why did you choose to allow this to happen, when it has now clearly resulted in yet-another-way to game the playing field?"

Just added now

There's a great deal (thousands) of whitehat webmasters who are being penalised to non-existence (or the effective equivalent). Sites that have demonstrated good practice, satisfied audiences and neither betray or mislead users. This is unacceptable, what are you going to do about it?

Following on from your statement confirming 'brands are not favoured': High levels of instability favour only those with 'deep pockets'. Given the current, exceptionally unusual, and well documented turmoil hitting some 'whitehat webmasters', isn't that indirectly favouring 'brands'?


I'm sure there are better, more useful ones - those were just first attempts. The point is it would be interesting to get real answers to these. REAL answers, beyond the usual 'disavow tool' rhetoric which is infuriating given the circumstances.

Fighting back with a 'chance'. If enough set this as a goal, in a few weeks we'll *probably* have answers. If you're not sure, then I say: It's better than anything we have now.

Also, there are a lot of eyes on this thread. (I've been in contact with bloggers who are 'waiting to see how it goes') - if it doesn't lose focus, we stand a chance of getting some momentum.

Our first shot should be 10-15 solid relevant questions.

[edited by: hitchhiker at 10:19 pm (utc) on Jun 16, 2013]

[seoskunk]

I believe 'Negative SEO/toxic links/disavowing' should have been something you vehemently guarded against. (The concept of any link or relationship as being 'negative' rather than 'ignored') No external entity should have been able to affect our websites negatively. Why did you choose to allow this to happen, when it has now clearly resulted in yet-another-way to game the playing field?

I think I can answer this, by simply ignoring links google actually encourage link spam. Since the greater percentage of links you have the greater chance they "count". Google's algorithm is based on natural links provided to good quality websites. Without negative factor on links this simply would become a competition or race for the most links and play completely in the hands of BLack Hat Seo. So a negative pagerank I believe was introduced that caused a dampenning factor on inbound links. Link velocity was also measured with sites gaining too many links too quickly raising a red flag.

Unfortunately BH started gaming the system and mixing authority sites in there link building networks. Negative seo was the result of penalisation of outwardly bound links. So the whole thing is a mess. If you ignore outwardly bound links it encourage's link spam, if you penalise outwardly bound links you encourage negative seo. I think personally google should be looking at link velocity and natural occurrance of links in market sectors (maybe they already are their smart cookies).

[diberry]

What the h*** else can we offer them?

Well, I hate to say it, but: "answers rather than questions."

You're talking about a dialog in which we basically ask, "How can I fix my problems, or why are they happening?" How does it profit Google to answer that? As others have noted, it may just be helping the spammers, and it's time/energy they could put elsewhere.

I'm not sure what we can tell Google that they don't already know. But maybe we could write something intelligent on why it makes more sense FOR GOOGLE to concentrate wholly on ranking content quality instead of counting links. Maybe we could host debates about what various user metrics really indicate. The sort of discussions they probably already have around the Plex, but we might manage to offer a fresh perspective.

[hitchhiker]

Hey man, thanks for that. I re-read it twice to make sure I understood you accurately.

It would be true, except you've not considered one major factor:

It is extremely dangerous, under any circumstances, to endow SITE B power over SITE A. Until this point, you lived or died by your own hand.

After this point they immediately introduced an entire 'realm' of new blackhat techniques. Far greater (it turns out) than the SPAM they were trying to conquer. It must be reversed.

@diberry - assuming 'they know' is (in my humble estimation) a mistake. Plus you are discounting the effect 'publically analysed mistakes' has on a large corporation.

we might manage to offer a fresh perspective

You have no idea how true that is, if it was possible I would inject you and a few others here directly into the plex and happily be done with this whole thing. Source: I've worked with one of the largest tech companies in the world (won't say who, not 'search', but they were probably the biggest, now not so much because of almost exactly this problem)

Core teams lost site of the bigger picture, their jobs were extremely hard, they were hit from all sides, they didn't organise efficient human feedback and instead relied on data sets to get feedback, those data sets (increasingly inaccurate as models grew) eventually capsized the ship.

[edited by: hitchhiker at 10:49 pm (utc) on Jun 16, 2013]

[TypicalSurfer]

There's a great deal (thousands) of whitehat webmasters who are being penalised to non-existence (or the effective equivalent). Sites that have demonstrated good practice, satisfied audiences and neither betray or mislead users. This is unacceptable, what are you going to do about it?

First they came for the blackhats, and I didn't speak out because I wasn't a blackhat...

[seoskunk]

Our first shot should be 10-15 solid relevant questions.

I would narrow that down to 5 questions

I once knew an seo that charged $1000 dollars a question. That's a bit expensive his client asked. "Yes it is" said the seo "so whats your second question ?"

[johnhh]

Well, I hate to say it, but: "answers rather than questions."

Well I have one answer - an easy fix for Google as well

Just take out all this domain crowding.

Search engines are about supplying choice to the user, based on whatever algo you like ( or don't like ), pages of results all with the same site ( or site.com, site.in, site.co.uk, site.otherTLD's listed ) are not really helpful.

Did one search this week 3 pages of basically the same domain.

Other searches I did showed similar results, one site listed 4-5 times on page 1

So instead of being on page 3-4 smaller organisations would move up. I'm quite a big fan of being first on page 2 these days.

I don't think you will see GoogleGuy or anyone else from Google on here any time soon, those days are well gone. I hope I am wrong but I think it's all PR spin now.

[Awarn]

@hitchhiker we all have questions. There is no use making questions until somebody actually has the power to contact somebody at Google in the know. I seriously do not think Matt Cutts is in the know. He is more a puppet PR man. So my question to you is do you directly or indirectly have a contact in the know at Google?

[diberry]

hitchhiker, you're right. Even when big companies have all the brain power they need, they also tend to have some closed minds and egos that get in the way. Sometimes hearing it from another source can make a difference.

Johnhh, domain crowding is a big question for me, too. I can't find anybody who's pleased to find multiple results from the same domain when they search, and I've been asking. So if it's not for users, why is it there?

[johnhh]

diberry - I have no idea, it doesn't happen for every search ( in my neck of the woods ) but enough times to be annoying. I have a funny feeling it's either a by-product of one section of the algo or the 'whitelist'.

[ The 'whitelist' may not be an actual list ( although I think it is ) but just a group of brands or favoured sites that heavily meet the algo's requirements, possibly by design. ]

Removal of domain crowding would please a lot of people, including me :)

[Savanadry]

Some excellent points in this thread. This can be summed up by google has become too powerful and has lost the plot.

Why is a search engine dictating to the WHOLE internet how they should link to each other, how often they should update their pages (never seems optimal at the moment), how many images they have above the fold and a zillion other things I'm probably not aware of but if I do them...I'm considered 'bad' and having done something 'illegal'?

Go back to the drawing board google, it is YOUR job to find the good sites, it is YOUR job to build an algo that works - WITHOUT IMPACTING THE INTERNET ECOSTRUCTURE.

Maybe a change of personnel, some fresh blood? A fresh perspective is what is needed?

Panda / Penguin has unleashed an almighty torrent of spam on the internet. I've been building websites since 1994, even in the days of AV and Yahoo it was never this bad. Good sites penalised for 'god knows what', spammers delightfully saying on forums 'go brand or go blackhat, and if you go blackhat go HARD - churn and burn'. They are making a fortune atm.

We shouldn't be in a situation where webmasters feel under seige, we have google one side limiting what we can do with our own sites. On the other side we have blackhats and spammers targeting our sites with crud that we have no idea what to do about. In the middle we're just trying to build businesses around a good service.

Google created this situation making negative SEO possible and rife, from my small corner of the internet it's getting worse month on month.

Bing doesn't have this trouble, they don't have 500 updates a year, don't dictate to webmasters what they can and cannot do beyond the obvious, and still provide an excellent service.

Google could learn a lot from Bing. There was a time when there were rumours that Bing copied google's results. Lol, they wouldn't touch the G results with a bargepole now, Bings are far better. What happened google? Get a grip or prepare to be ousted by the people who REALLY DO run the internet, the millions of decent webmasters and users who make the internet such a wonderful, varied, vibrant place.

[bobsc]

Google could learn a lot from Bing. There was a time when there were rumours that Bing copied google's results. Lol, they wouldn't touch the G results with a bargepole now, Bings are far better. What happened google? Get a grip or prepare to be ousted by the people who REALLY DO run the internet, the millions of decent webmasters and users who make the internet such a wonderful, varied, vibrant place.

The USERS run the internet.

Google is a bad habit.

The USERS need to break the habit. Intervention!

[hitchhiker]

@Savanadry: Thanks for taking the time to convey your situation, given your considerable experience and perspective.

We need to see MORE highly experienced, informed webmasters taking the time to post about what they see going on.

If you think it doesn't affect you yet -> perhaps it's not wise to wait until it does.

[Umbra]

The USERS run the internet.

Google is a bad habit.

The USERS need to break the habit. Intervention!

I did it. I switched.

[hitchhiker]

Some freakin' genius has cleverly injected a full on SEO 'proof of failure' into the SERPs.

Read about it here (at the end, just before I'm calling the dude a king): [webmasterworld.com...]

Screenshot: [imgur.com...]

[diberry]

I think - just based on what I see in my own searching - that Google tries to index the whole web while Bing is content to cherry pick. Therefore Bing has the better quality results, but Google has the more comprehensive. The problem for Google is that (I suspect) it's just not possible to index the whole web in a meaningful way anymore, so they're struggling.

LOL @ the proof of failure.

@Savanadry, I started building sites in 1998. Back then, we linked as we saw fit. We even - gasp - PAID/traded for links from sites whose audiences were likely to enjoy our site. It was all about referral traffic, and in my mind it still is. The first thing Google ever did that I strongly disagreed about was to punish people for paid links. I get why they did it, but I think it was the wrong approach. And now we've slipped on down the slope to where negative SEO works because of the link fiasco.

[heisje]

@ Cutts payday loans:
Proof of the pudding is in the eating!
Verdict : Miserable failure.

[HuskyPup]

Like several others here I'm coming up to my 20th anniversary of SEOing, and yes, it did exist back then despite what some may believe!

There are some excellent points in this thread that Google as a company needs to address and especially so IF they expect US to co-operate and supply information free of charge to their "knowledge engine".

First of all we deserve their respect for ALL the good work we've done and from which they have earned money, let us not forget this point, without US Google would not exist ... Well, unless it only wanted to display Wikipedia, Amazon and eBay ...

The shareholders also need to understand this too since it is a "flaw" in their business model, one of expecting us to keep providing this information for free meanwhile dividends are expected to be paid whilst many webmasters are struggling to even make ends meet.

There needs to be a fundamental change of attitude by Google towards many of its information/knowledge providers and especially so towards the niche widget sites. I, for one, am 80% down from two years ago and, believe this or not, 60% since January of this year alone when the new "image whatever" was introduced.

For me now it would not make any difference if I denied Google access to my sites. At a recent very big widget trade fair in Germany I was repeatedly told that businesses were not using The Net to locate new suppliers since the results were simply awful and not credible. Amazing, it's come full circle, I'm back to face-to-face selling again.

I have one other thing to relate to whatever has happened at Google and I am totally confused by it.

Whilst in Germany I confronted the webmaster of a site that had completely scraped one of mine with many of his pages, and especially so images, having replaced mine with most of them ranking #1. His sites too had been hit by whatever's been going on, in fact he showed me his live data and he has been hit harder and my site is doing 5X his traffic!

This has left me confused, is my widget sector really doing THAT bad, have so many people stopped searching for my trade widgets? Whilst I am one of the major half-a-dozen global brands in my industry Joe Public doesn't search for my brand simply because they don't know it, however the trade come to me direct.

So ... I've lost a lot of search traffic but also the major scraper in my trade has done so, as well, yet he's still ranking at the top.

Confused? You bet I am yet Google is the only one with the answers for me, tell me Google, just had bad is the construction industry doing?

[totalodds]

If everyone can take a few minutes to Tweet @mattcutts with the URL of this thread, I am confident he will read it.

Take two minutes of your day to Tweet & get other webmasters to ReTweet.
Get the ball rolling on transparency.

I've just tweeted him, done my part :).

[Bigwebmaster]

Hitchhiker - Just wanted to quickly say this is an awesome thread that you started.

For all of you simply Google Bashing, it doesn't help anyone. You have already done this in countless threads already over the years. Has anything changed from the result of Google Bashing? Nope.

Back on point, I also really think some sort of channel needs to be opened up from Google to webmasters who run website that meet or almost meet Google's Quality guidelines. Skip the webmasters who are in clear violation, just try to work or figure out which webmasters may have been collateral damage, and then start with them. In the past I would have said that this probably wouldn't be necessary, but with updates like Google Panda, Penguin, and the lesser known Ghost Update that can literally wipe a site off the face of the planet, I think it might be a good idea if Google is wanting to look at the long term and make sure quality websites like this continue to be developed.

I can understand Google's reluctance to want to open up any direct communication because there are just too many websites, and this could overwhelm them. However, I think they could have ways to keep this more manageable by maybe first starting with websites that are being hit that are in the grey zone. This grey zone could be a little or as large as they want, but sites that are on the wrong side in the grey zone would be the webmasters that they might open up a dialog with via Google Webmaster Tools.

If they really don't want to dedicate any personnel to this, they could always automate this and point out to these websites what the main factors are causing them to be in this grey zone specific to their website. Or if Google wants to be careful on who they help, they could allow these sort of websites to be quickly reviewed by a real person to make sure the website really does fit Google's Webmaster Quality guidelines, and then if so, feedback, whether in an automated matter or not, could then be provided to help that particular website get out of the grey zone.

[Bigwebmaster]

totalodds - I did my part and tweeted this URL to Matt Cutts.

[heisje]

Has anything changed from the result of Google Bashing? Nope.

Fallacy!

Google, a company that once used to be the "darling" of the 'net, has gradually been exposed by your so-called Google Bashers for what it really is : a vicious monopoly, grabbing whatever may be grabbed, amassing a fortune to the detriment of everybody else, by employing methods of dubious legitimacy and certainly dubious ethics.

Awareness that did not exist 5 years ago.

At exactly the same time that some rather brave souls (much ridiculed most of the time, as also today by you) were working to expose reality, others, in parallel activity, were trying cowardly & slimily to appease the monster, in effect begging for their lives.

What have the results been of their cowardly behaviour during the past 5 years? None. The monster has eaten them alive.

“All truth passes through three stages.
First, it is ridiculed.
Second, it is violently opposed.
Third, it is accepted as being self-evident.”
- Arthur Schopenhauer

I am not against the currently proposed effort, as it cannot do any more harm - the situation is already bad. I am just urging that reality should not be forgotten or ignored.

.

[edited by: heisje at 7:01 pm (utc) on Jun 17, 2013]