7 May, 2013 - My Most Important Pages Have Vanished from the SEs

[sugarvalley]

To begin, my heart goes out to anyone who was hurt due to no fault of their own by the recent algorithmic changes.

Things turned negative for my site on May 7th. Prior to that, my rankings were decent (pages 1 and 2) for about a dozen high traffic keywords. On the 7th, the high traffic keywords dropped completely, while the long tailed keywords (usually went to less important pages on the site) held steady.

Unfortunately since the drop, my site (9 years old, very well optimized, no spam, no ads) has suffered from a massive drop in traffic/sales - so much that it is stressing the heck out of me. I never engaged in spam, posting links on chat boards/forums, linking schemes, etc.. In other words, no unethical behavior. The most I ever did was the occasional guest blog post on sites that were somewhat related to my industry. Checking my inbounds, there appears to be very little spammy looking sites linking to me. Most (90%+) of the sites I have never seen before and they are linking to me without my knowledge or approval.

To summarize, the most important pages on my site have vanished from the SEs, while my less important pages continue to rank decent. I am scared because this is how I make a living, and without the orders, I'm dead in the water.

[edited by: tedster at 8:56 pm (utc) on May 24, 2013]
[edit reason] moved from another location [/edit]

[GeraniumV]

Sugarvalley - funny you should mention the May 7th date - I was looking into my traffic after the Penguin update and I found something of a mystery. I've had a huge shift in traffic from my larger sections to the smaller sections of my websites. It appears to have begun on the 7th/8th and seems to be ongoing daily. My largest two sections seem to have lost close to 30% of Google traffic, but for the most part I've seen huge gains on smaller parts of my site.

Is the traffic loss spread across your entire site?

[edited by: tedster at 9:35 pm (utc) on May 24, 2013]
[edit reason] moved from another location [/edit]

[sugarvalley]

GeraniumV - Basically I'm off the radar for high traffic terms that went to certain pages (including the homepage). However, for long tailed keywords, I've held steady. Unfortunately, due to the loss of ranking for the important terms, traffic is down at least 75%. For the lesser important pages on my site, traffic has always been low, so while they're ranking about the same as before the update, it's not enough to keep my business afloat.

Regarding the recent 2.0 update, I'm guessing that Google doesn’t like who is linking to me for the keywords that I'm being penalized for. What's so frustrating is that I have evaluated my inbound links and see very few spammy sites linking to me. The ones that look spammy are Chinese directory sites, some Russian, etc. Most pages linking to me are ones that I've never had anything to do with. It doesn’t appear to be bad enough that it would have hurt me, though I’m probably wrong.

One thing to mention is that two of these inbound links came from blogs where my URL appeared sitewide in the 'blogroll' with the penalized keywords. These two blogs were run by an acquaintance of mine from 3 years ago. I wrote him on Monday, the 13th, and asked him to please remove these as they look unnatural, and he did right away. After thinking everything over, this could be the reason why I was penalized for only the certain keywords. And if that's the case, I need to follow the recommended procedure for getting back into the index. (or find new keywords?)

[mrguy]

To summarize, the most important pages on my site have vanished from the SEs, while my less important pages continue to rank decent. I am scared because this is how I make a living, and without the orders, I'm dead in the water.

If your basing your living on Google's free traffic, you should use this as a wake up call.

The traffic they provide can be here one day, and gone the next.

[tedster]

I'm certain there was some kind of Google Update on May 7. So far, their spokespeople have been silent about it when asked, but it seems clear it was not some kind of "early Penguin" test.

You said your pages have vanished "from the SEs". Does this mean your pages also vanished from Bing? Also, do you have a Google Webmaster Tools account set up?

[GeraniumV]

What I am seeing is May 7th/8th USA (based on PST), May 9th UK (although not as large and the UK site stabilized on the 20th and got a boost on Penguin) - I just analyzed my UK site.
I don't know if I'm looking at the same thing with the UK as it spiked on the 8th before the change, and the changes seemed to subside on the 20th.

[sugarvalley]

@mrguy - You're absolutely right - this has been my wake up call. All this time, I was relying on Google organic rankings. It's forced me to expand my marketing strategy to include Bing, social media, and possibly hiring independent contract salespeople. Besides, since the update, the 1st & 2nd page are almost completely occupied with major retailer sites, making it impossible for a small retailer to compete. The 1st page of the index now reads like a bad advertisement. It's almost like, the bigger the company, the better the ranking...

@tedster - It was just with Google. Bing ranking is fine, although I'll need to work on improving it some. Yes, I have a Webmaster Tools account. Not sure whether to use the disavow tool for the spammy sites or file a reinclusion request? I know of someone who filed a request about a year ago due to a bad linking campaign (forum spam i believe) and within two weeks, his site was back on top.

Regarding the update around the 7th, the only article I could find that mentions it is If That Was A Google Update You Felt, Google’s Not Confirming It [searchengineland.com] - featured on another site.

[edited by: tedster at 4:50 am (utc) on May 25, 2013]
[edit reason] Added the link [/edit]

[chrisv1963]

Same drop in Google Traffic here, starting on May 7th/8th. This is for my good website with a lot of original content and updated almost daily.

Weird: At about the same time pages from an old rather crappy website I own (Not updated in about 8 years!) started showing up again on the first page of the serps.

This must be some "quality penalty". If the quality of your content is too good, then it is clear that you are trying to get good rankings => seo spam => penalty! :-)

[ecmedia]

I can also confirm that I have also seen drop in some of my websites around this time. Like everyone else here, these are good websites that have been there for more than a decade. It looks that if you are white hat, Google punishes you repeatedly while spammers thrive.

The funny part is that without getting very sophisticated, when I simply search my keywords here in America, the rankings are better than ever. So I wonder if I am losing overseas traffic. I also notice that the Adsense CPM is higher than ever -- meaning more US traffic -- and that has tempered the impact on income.

During last few days, I am noticing a small recovery.

[LuckyLiz]

We had a big drop in one of our websites starting May 8. Each week since then it seems to have dropped a little bit more.

The site is updated regularly, and never engaged in any kind of link building campaigns. It does get promoted in social media, though. It also has always gotten a lot of links naturally from credible sites. But webmaster tools shows a bunch of sites linking to us that might be spammy, and that we had nothing to do with.

WikiHow comes up often as the first result (sometimes both the first and second result) for some keywords we used to rank in one of the top 2 spots for

Our main topic index pages have just about disappeared from SERPs. The first page of SERPs for various keywords associated with the topic seems to include many more big company brands than they used to.

Interesting, is that when I look at other sites in this niche that are not owned by big businesses, their traffic (at least what's visible in Alexa) have fallen in the same way. So to have other informational sites I know of in other niches that are privately owned and have good content.

Maybe the May 7/8 update should be just called the Big Brand and Wikihow update......

[brianp]

Okay, having seen this same issue hit us on 5/7 I've been digging around for potential causes.

Background: 10+ year old website, e-commerce, unique industry and Amazon and Ebay do not participate in. Hit bad with Panda 1, have been increasing in ranking, traffic, impressions, and conversions since we started working on addresses Panda issues.

The only thing we've done really in the past year is continuously add new products to our website. Running Magento Enterprise, and we use original content that is well written and consistent after the initial Panda.

What I've found so far:

1) We have some issue where Magento's URL rewrite process has went crazy with specific products. It appears to be ones that shared URL keys, where Magento made the next one be widget-type-####.html whereas the first one was just widget-type.html. I've found on average 200+ url's that were created on each product. Rel=canonical is turned on, yet each of these mistake created products through the URL rewrite process has it's own rel=canonical for its URL.

2) We had around 120~ internally linked pages that were returning 404s.

We've addressed #2 and we're working on cleaning up #1.

The interesting thing is, some how Google has found around 54 of the URL's that were generated incorrectly through the Magento URL rewrite process. I do not see them in sitemap we submit, and Screaming Frog did not find any in the scan I did. So it is a big mystery as to how Google found these URL's, why they only found 54 out of the thousands.

I hope that helps someone. Basically my advice would be keep looking, there is most likely a situation like we had, where a problem you wouldn't normally see has been found by Google and it is hurting your rankings.

-BP

[brianp]

A few details I did not include:

We saw a HUGE spike in crawling activity the week before this update hit. 30,000+ pages crawled per day, when our website has maybe 2500 total.

Searching Google for some of the mistakenly created URL's definitely shows many more were indexed that weren't listed iN WMT.

If I had to guess, I'd say the 5/7 update was perhaps when the adding Panda to the main algo was implemented. Everything I've found thus far looks more Panda like than anything else.

[krikomoush]

I'm certain it is panda on 5/7 we were hit on our pandalized sites, and those sites have not ever had any other algo issues.

[tedster]

But Panda is now real-time integrated into the rankings. There are no updates at specific times any more.

[GeraniumV]

sugarvalley - I looked at the SERPS thread here and got to May 9th and discounted that date - as I was seeing most of the drop on the 8th and 7th - most of the damage was done well before the 9th on my US site.

As a note both US and UK have almost similar structures - but the UK site seems especially sensitive to algo changes.

Looking at the US site as a 2 dimensional tree - traffic has shifted right and up.

Like you I get most of my traffic long tail, but see higher level keyword phrases popping in and out. The sections hit are probably over optimized, have more backlinks (because they are obviously larger) and really verbose description tags (I probably have the worst description tags in those sectors).

The other things are repetitive text in the anchors and perhaps the length of the text in the snippets I show in the listings (because some traffic seems to have migrated to the twigs on my tree). I changed one section to use distinct anchors, and increased the length of the snippets.

Does anyone know if the 7th and 9th updates were country related?

tedster - I wouldn't discount a Panda update - in the old days the collected the data and then ran it. In real time you would think a change would run on the data on hand and then apply those changes as additional data becomes available.

[TheOptimizationIdiot]

...in the old days the collected the data and then ran it.

There's nothing I've seen to say they aren't doing the same thing now.

They've said they're running Panda automatically on the front-end of rebuilding the index rather than "in addition to" an index rebuilding cycle, but that's all they've stated has changed.

So, basically, the Panda running schedule has been "bumped up" to whatever time period they rebuild any given portion of the index at, but that doesn't necessarily mean the data used or the time period of collection has changed, it just means when/how they run it is automated relative to any portion of index rebuilding rather than running Panda manually in addition to the current index rebuilding.

So, for example, if they used 3 months worth of data collection for running Panda separately, there's nothing I've seen to say they don't still use the previous 3 months worth of data collecting when Panda is run in front of an index portion rebuild. They could (likely imo) still use the same time period of data collection, but the data would "roll through" the portions of the index sooner for queries updated daily (or in real time) where it would "roll through" more slowly on portions of the index rebuilt less frequently.

Example:
News portions of the index are likely updated in near real time (QDF), so "now minus 30 days = Panda data", but "stale is better" searches may be updated less often, so "now minus 30 days" could seem to be based on something different, while still being based on the same time period of data collection.

9 years old, very well optimized

Could you please define your idea of "very well optimized"?

I ask because, "very well optimized" changes over time based on "changes to the rules" and what was "very well optimized" a few years ago isn't the same as "very well optimized" today, so I guess I'm wondering mostly if you have adapted to the changes in what "very well optimized" is over the years or if your site is still optimized like it's 2004 or 2008?

[edited by: TheOptimizationIdiot at 1:55 am (utc) on May 28, 2013]

[GeraniumV]

"...in the old days the collected the data and then ran it."

I might be confusing you - I was talking about Panda - "in the old days" it happened and it was over in an hour or so.

Whatever happened on the 7th,resulted in a big change but it didn't finish there (at least in my case). I was presuming if something took a lot of processing power and was in real time there would be after shocks as new data propagated through the system.

[TheOptimizationIdiot]

I would think that would depend on the update schedule for a specific portion of the index. I personally think they update different portions of the index at different intervals to save on processing cycles/computing power necessary to update the entire index.

Also, I was editing and got distracted while you were posting, so see my previous post for an example. (Sorry, life got in the way of a post, oops! lol)

Another, possibly better, way of explaining what I'm thinking would be if a portion of the index is updated more frequently (near real time), there could be "days of bouncing" based on historical changes made to a site over 4 or 5 or even 10 days, yet for other portions of the index there may be no "bouncing" due to less frequent rebuilding of that portion of the index.

So, in one case, on-site (or whatever) changes over 4 or 5 or 10 days could have a bouncing effect for that number of days, but in other cases those on-site (or whatever) changes may all be "lumped together" in the next update of that portion of the index. (Hopefully that makes a bit of sense. I know what I'm trying to say, but sometimes I have difficulty in spitting it out lol)

[TheOptimizationIdiot]

Another addition:
If a portion of the algo takes a large amount of processing power to generate and it's run on the front-end it might take longer to build that portion of the index, but that doesn't necessarily equate to "you should see bouncing" based on the data collected and used for "publication" of the index, because if the data is accumulated and integrated into the algo the "publication" of the index could be delayed, but if the next "publication" is not for another 7 to 10 days, then there may or may not be "bouncing" depending on what the data accumulation over the period not already accounted for is.

A shorter version is:
When something takes more processing power it could (would likely) delay "publication" of a portion of the index to some extent, but "delayed publication" is not necessarily a precursor to, or indicative of, "bouncing" in the results for any period of time. (Still hoping I'm making a bit of sense to someone besides myself LOL)

[GeraniumV]

Hi, after doing more analysis - I was hit late on 7th/8th - but that's nothing compared to the real problem.

The fall on my main sections has been happening for a while - I get a seasonal drop - so it wasn't a worry at first.

Here's the history:
In April I deleted some sections which were basically thin affiliate stuff, and beefed up some smaller sections. The beefed up stuff is doing great.

Unfortunately I made some changes to the main section that I thought were minor but have been a disaster.

1) I took some of the affiliate products from the deleted sections and turned them into rotating ads. Unfortunately I had a typo instead of re="nofollow", I had re="nofollow" that can't be good - I only found it because I used one of the W3C verification tools. That's fixed.

2) I also screwed up the rewrite for the main sections by moving a line of code to support my small sections which resulted in the /directory not been redirected to /directory/ but only for my main section. I found those buried in the WMT errors (I had a lot of 404s from the deleted sections - so I wasn't really looking). Cleaned up.

3) To top this all off in the footer link of the important sections I also had a typo for a url "directory/" instead of "/directory/" - which has generated an extra 404 for every page in that section. Cleaned
I read this thread:
[webmasterworld.com...]

My question is what to do with the 404s.
1) I could just leave all alone and let Google rebuild the map.
2) I could put back the deleted sections and NOINDEX.
3) I'm not sure what to do with the phantom pages 301 them back to the root page or just let the 404 play out.

Yes I feel like a dumbass.



and am wondering how to proceed

[Andem]

Unfortunately I had a typo instead of re="nofollow", I had re="nofollow" that can't be good

I suggest you take another look. You just made the same typo. It should be

rel="nofollow"

[tedster]

GeraniumV, nice job on being persistent and finding those errors. Celebrate the vistory - many people fo not look that hard.

[reseller]

Celebrate the vistory - many people fo not look that hard.

@tedster

Pssssst....

What did you mean by vistory and fo :-)