Google Traffic Dropped over 60%, Nov 17th 2010

[alorentz]

We are a leading publisher of how-to marketing information with over 380,000 subscribers, and approx. 1mil pageviews/month. Our site is content rich, SEO'd, we have hundreds of thousands of back links and we've been very well indexed for years (new pages are usually indexed within 15 min). Businessweek, and some others, actually solicit membership for our content.

On Nov. 17th 2010 our Google site traffic dropped by over 60% and has not rebounded.

I've also noticed that our indexed pages, including home page, have fallen farther back in search results dramatically (typically fixed to top position on page #2 in results, if listed at all) for some of the regular searches I do to check results. I've seen some opinions on the possible causes, ranging from Holiday SERPs, Google Preview/Instant, other algorithm changes, etc., however, these are all guesses. I don't believe we are violating any Google policies.

A couple of things I've noticed:

1. Our content is scraped daily, either in full format or excerpt, and these feed-based sites are ranking better than us, the source, only after Nov 17th. We normally ranked #1 for our content, however, something changed, and we are now often demoted to page 2 or not shown at all, despite being the source.

Our content still gets indexed within a few minutes of publishing, however, searches for our exact content titles show sites that are pulling our feeds or scraping our content, instead of showing us.

We used to be listed #1 for content title searches, or anything closely related, and now we are farther back in search results if at all, superseded by feed based sites. And all our keyword searches have dropped, along with traffic.

Some searches result in the Google message "In order to show you the most relevant results, we have omitted some entries very similar to the 1 already displayed. If you like, you can repeat the search with the omitted results included."

Scraper sites are listed for this content, and they scraped it from our site in full. If you click the "show more" link it then shows our site. This also leads me to believe that Google has lowered the rank of our site pages, for some reason, as we are now thought of as the duplicate site.

Many, many sites illegally scrape our content, and we can't track them all down. Most of these sites are completely bogus and just used for Adsense or advertising purposes.

How in the world does this happen? How can't Google determine source site and site quality. Site that are ranking are simply scraped/feed-based sites, and we've been a top online marketing publisher for over 10 years.

I sought feedback on Google Webmaster Forums, but am hoping to get additional input here.

Thanks for your time and consideration.

[Sgt_Kickaxe]

I know of 3-4 large sites that would fit your description and they cover a wide range of subjects on each site. 3 of these 4 have "borrowed" my content and re-written it slightly as well as "borrowed" a few dozen of my images, for my subject of choice.

I'd be willing to bet Google is de-valuing sites with a higher % of "borrowed" content. If the site name starts with the letter E I'd say the guides are rather short and crude and heavily monetized, not exactly the quality Google is looking for. It may be that small but highly focused sites just got a leg up on massive aggregate sites. That would be a reversal from last March when mashup became king.

re:#1 - improvements to your link structure can overcome this if your site isn't too big.

I'm not suggesting your site is a scraper site but look closely and see if it quotes many other sources, I bet it does.

[alorentz]

@sgt_kickaxe - we do reference other sites regularly, as well as publish original content. All content is written by our paid authors, even when referencing other sites.

On that note, if we are being devalued for referencing other sites, how is it that the sites that scrape us (or pull our RSS feed) are gaining better ranking for our content. Most of them do point back to us as the source, so I would think that would benefit us. However, it does not appear so.

I've also started to include "original-source" meta tags, to signal that we are the true source, although Google has said those tags are just for testing and don't impact rank at this time.

[alorentz]

@sgt_kickaxe - your point is interesting though. Is there any information published to support this belief that Google is frowning on aggregate sites?

[jimbeetle]

There have actually been a few articles in the mainstream press over the past month or so on the subject. Nothing to really say yea or nay that any SE is actually doing anthing about it at the moment -- though your site's situation might be telling -- but once this stuff goes mainstream you know it's on the SEs' radars.

[Planet13]

Are the sites that are scraping your content and ranking higher than you more narrowly focused than your sites? Or do those scraper sites also cover a large area of content?

Have you noticed different sites rising to the top of the SERPs over a short period and then dying off, suddenly? Or do they rise and stay there for a while?

[alorentz]

There are far too many sites to determine how they each position in SERPS (hundreds, if not thousands). We have over 20k subscribers to our feeds (that display excerpt on their sites), plus those the scrape. Anybody can build a WordPress site completely built on feeds, and become well ranked.

As a test, I did just that a couple weeks ago (build a new site that pulls our feed or excerpts), and the new site is already outranking our primary site for some searches.

Specifically, I'll see sites like Tweetmemem, Twitter, OpenPulse, and others, that simply list the content title and link to us, and these are also outranking us. Most of the site are simply title and teasers of other sites, yet they are becoming the top ranked sites. If Google is frowning on sites that aggregate, I would think these sites, with literally no original content, would receive lower ranks.

I would also think Google could do better to identify the true source, which we are. Again, we have hundreds of thousands of backlinks to our content.

[alorentz]

If there are any specific articles on this issue, related to highly focused sites outranking larger sites, I'd love to read them. If this forum is not the place to post those links, perhaps sticky mail me (if that is acceptable)

Thanks.

[tedster]

It is completely acceptable to post links to articles from the mainstream press. My sense of this is that Google has been adjusting the algo since the infamous Mayday Update [webmasterworld.com], working to find a balance particularly on long-tail searches.

The balance Google is chasing is "trusted sources" or brands - versus pinpointed relevance even from a smaller site. In other words, good results without too many bottom feeders, scrapers, etc. Along the way, I'd say there have been several attempts that generated a lot of false positives against honorable sites, and also lots of spam and scraper sites that slip through the net.

[alorentz]

Yes, that's the big question ... will our site be reinstated to previous rank from Google revisiting their algo change, or have we now been devalued as duplicate content (despite that not being the case). At this point, I'm trying to determine what the problem could be, so I'm not chasing my tail changing things that have no impact. :) However, it seems the only true answer, specific to our site, could come from Google.

[alorentz]

Note, I'm not asking this forum that question, it was more rhetorical. Anyway, any articles on this subject specifically are appreciated.

[bwnbwn]

alorentz I would dig deeper than the scraped sites for a cause. It could be that but I really think there is something else. Just about every site on the web is scrapped and republished. I do know this when I was filtered about 3-4 years ago this is the same exact thing I saw in the serps. I have a feeling your site has been filtered for something.

[alorentz]

@bwnbwn - I've done the due diligence on my end, fixing 404s, removing UGC spam from comments, nofollowing UGC links, canonical tags, optimizing pages, etc. We have over 200k pages indexed, and normally have new content indexed within 15 minutes (however, not ranked well anymore). There are many pieces to this puzzle. I realize there is no silver bullet, I'm just unclear on what else I can do, or even determine if our site has been "filtered", and/or penalized (haven't received any messages in webmaster tools though).

The idea that we could potentially be thought of as an aggregate/duplicate site (from Sgt_Kickaxe), despite having a significant amount of original content, is an interesting one though. Again, though, there doesn't seem to be any way to confirm any of these theories. So, in the end, we potentially end of changing a lot of things that grow our business (adding cost and making it more difficult to grow), when they may not be related to the problem at all. Know what I mean?

[pageoneresults]

On Nov. 17th 2010 our Google site traffic dropped by over 60% and has not rebounded.

Any technical changes recent to that date? And, are you absolutely sure? Did anything change with a rewrite rule? Maybe a simple tweak here that caused some fallout over there? Just wondering, whenever I've investigated large drops in traffic, most were related to a technical mishap.

I'd be looking at crawl activity before and after Nov 17, 2010. That would be a clear sign of something not right. Usually a spike and then little to no crawl activity afterward. In one of scenarios I've reviewed, there was a rewrite rule that was modified and full testing was not performed. Thousands of pages went 404 without the owner knowing it.

After a quick check of GWT, the "signs" were there. The spike in crawl activity which was the bot choking on all the 404s and then little to no crawl activity after that. Once fixed, crawl routines returned to normal and so did everything else.

[alorentz]

@pageoneresults - The first assumption was that something was wrong on our end, and anything relevant (mentioned prior) was fixed over 4 weeks ago. I've also sent reconsideration requests (although we weren't de-listed). Basically, on Nov 17th, site traffic dropped, not progressive in any way. Nov 16th we had over 13k visit, Nov 17th, only 3k, and all reports in GWT and GA showed a drop off that has not rebounded. Traffic before Nov 16th, was steady for years, no spikes, no odd drops.

[bwnbwn]

due diligence on my end, fixing 404s, removing UGC spam from comments, nofollowing UGC links

Can you tell us how many were present beore you did this, and if there are more live links have you verified the site to make sure it isn't a bad link or lands on a malware loading site. I understand you should have been notified through WM area but it could be a possible cause.

What your describing to me sounds and smells like a filter. I would look over the server log files about a week or so before the hit if that is possible.

[alorentz]

"I would look over the server log files about a week or so before the hit if that is possible. "

Looking for what exactly?

RE: present prior.

1. 404: about 50-100, either from inbound links or UGC
2. SPAM in UGC: about 10 comments with explicit or prescription words (omitting specifics to not impact forum SEO :)
3. nofollow links added to entire forum (35k+ pages), however the forum is moderated for quality, so links should have been OK.

[tedster]

You might learn something by isolating the googlebot interactions with your server. Looks for the URL requests and the server response, see if the pattern changed.

[aakk9999]

nofollow links added to entire forum (35k+ pages), however the forum is moderated for quality, so links should have been OK.

Just a speculation:

So you had what you think were OK links and you have nofollowed it on 35K+ pages. Assuming only one link per page (but there are probably more?), this is at least 35K links turned from free linking out to nofollow. Pure speculation, but I am wondering whether some trust could have been reset by this? Especially if, as you said, these links were moderated so were OK for quality rather than spam?

[alorentz]

@aak - I'm not clear on what you are speculating. Do you mean:

1. Trust is now reset by adding nofollow, so it could help restore Google SERPs
2. Trust was reset by adding nofollow and potentially caused this drop?

If #2, to clarify timelines, the nofollow was not added to our forum until late Dec. The traffic drop occurred on Nov. 17., and would be unrelated to nofollow change.

[alorentz]

To add, in relation to the day Google traffic dropped, some SEs also had the same drop, and other did not. For example:

Yahoo and Bing had a seasonal dig Nov-Dec., but nothing out of the ordinary, and both of these SEs are actually driving more traffic day by days now (likely a result of our newsletter frequency change). Good, none the less.

However, Google, Ask, Search and AOL, all had the same type of mass drop off on the same day.

[indyank]

Did you mean 1mil pageviews/month or 1mil pageviews/day?

Guess you are complicating things by adding nofollow to all the outgoing links.

This month google seem to be ignoring nofollows as I see pages moving up with just nofollow links to them.It looked interesting to me, but does anyone else see this? I have a feeling that google determines whether to pass on the juice or not based on an algo of its own, rather than depending on the nofollow stuff and it is definitely possible for G to do this.

[alorentz]

@indy - 1mil pageviews/month.

[aakk9999]

If #2, to clarify timelines, the nofollow was not added to our forum until late Dec.

Sorry, I meant #2, but got your timings wrong, so this is not the reason of your drop then.

[alorentz]

@aak - Understood. Note, "The traffic drop occurred on Nov. 17., and would be unrelated to nofollow change." I just wanted to make sure you were clear on the timing. Thanks.

[pageoneresults]

Our content is scraped daily, either in full format or excerpt, and these feed-based sites are ranking better than us, the source, only after Nov 17th. We normally ranked #1 for our content, however, something changed, and we are now often demoted to page 2 or not shown at all, despite being the source.

That's a sure sign that your site has some challenges. If the scrapers are able to outrank your site that easily, there must be something wrong with the originating source. It shouldn't work like that if all your ducks are in a row (as they say). It is typically a first come first serve and then the authority takes the lead, if they were not there already.

Our content still gets indexed within a few minutes of publishing, however, searches for our exact content titles show sites that are pulling our feeds or scraping our content, instead of showing us.

Which means that the scraper sites have a stronger position than you do from an indexing perspective. Apparently you've got some roadblocks that are allowing the scrapers to outrank you. That really should not happen. I can see them getting on to the first page of results but not outranking the original authoritative source.

We used to be listed #1 for content title searches, or anything closely related, and now we are farther back in search results if at all, superseded by feed based sites. And all our keyword searches have dropped, along with traffic.

Is it possible you've come under attack by a competitor? It happens regularly, especially in specific industries. Maybe someone found a flaw in your platform and is capitalizing on that? Have you looked closely at those scraper sites? Backtracked owner information? Researched the "why"?

Some searches result in the Google message "In order to show you the most relevant results, we have omitted some entries very similar to the 1 already displayed. If you like, you can repeat the search with the omitted results included."

Scraper sites are listed for this content, and they scraped it from our site in full. If you click the "show more" link it then shows our site. This also leads me to believe that Google has lowered the rank of our site pages, for some reason, as we are now thought of as the duplicate site.

Many, many sites illegally scrape our content, and we can't track them all down. Most of these sites are completely bogus and just used for Adsense or advertising purposes.

You probably need to start getting a little more proactive in blocking the scrapers. I'd also suggest the use of noarchive and block Internet Archiver (IA) from accessing the site. That is "my" first line of defense. If you want it, come to the site and get it so I can record your activities. I don't want you surfing cache and scraping my content from there without me knowing it which happens day in and day out - the scraping via cache that is.

How in the world does this happen? How can't Google determine source site and site quality. Site that are ranking are simply scraped/feed-based sites, and we've been a top online marketing publisher for over 10 years.

It happens because the "source site" may have weaknesses that can easily be exploited when scraping and republishing.

[alorentz]

@pageoneresults - Understood. But those are general rule of thumb, and have been reviewed. This doesn't seem to be a simple SEO issue.

The scraping is nothing new. We've had it done for years, just more so lately, with the advancement of WordPress plugins.

And, at this point, I don't even know if scraping and RSS pulls are the problem. It's just one possibility, of the numerous possibilities for a drop of this type. There are certainly no flags helping to identify the problem.

All,
tedster recommended that I not post links in the forum, but would knowing my site URL help in any way? (if permitted to post)

[Planet13]

All,
tedster recommended that I not post links in the forum, but would knowing my site URL help in any way? (if permitted to post)

I would send a sticky mail to tedster and ask him. I know that in rare cases where it benefits the group as a whole, a URL is allowed to be posted.

[tedster]

Sorry, this isn't an exceptional case. There is a Site Review [webmasterworld.com] forum (subcription needed) in the private Supporters area where members can share links and openly discuss all manner of issues. Doing that in the public area generates too many issues.

[alorentz]

@tedster - no problem.

I realize it's hard to say how effective the Site Review section is, but what should I expect there? Meaning, if I pay for a subscription, I would hope to get more out of it than just going to Google Webmaster Forums, where you basically get general SEO tips, and templated answers they give to everyone that complains about their site not being indexed :).

I'm already pretty informed with SEO best practices, so the general "make sure there's no duplicate content" or "have you analyzed GWT data" suggestions don't really help. Our site is SEO'd well, but there's something more to this problem.

Thanks.