Stuck on the Road to Recovery from Panda Update - Need Help

[sriki]

I would like to share my experience and ask a few questions with respect to Google panda update that hit my site (5+ years old blog running on wordpress) very badly the day it was released on Feb 2011. My site lost about 60% of the traffic by the time panda went global. Here are some of the interesting points to notice when I started making significant changes:

1. As per the instructions from Google's official blog, I made a list of some of the posts that I thought of as low quality and deleted them.

Result: After a few weeks traffic dropped by another 5-10%.

2. I started to notice in the webmaster tools that my index status was showing 600+ when I originally had only 130+ posts on my blog. Meanwhile, I enabled reply to comments option from WP dashboard that created a huge problem. More than 3k ?replytocom links got indexed and contributed to duplicate content problem.

Now my index status in webmaster tools started showing 4k+ as indexed status instead of 130+ actual posts I had.
Initial traffic before panda update was around 5.5-6k unique visitors per day. But this dropped to around 900 after the ?replytocom problem occurred.

3. I installed replytocom redirector plugin that 301 redirects all replytocom links to original URLs. I also fixed all the duplicate pages that got indexed. After a few months all replytocom links got de-indexed and now the index status gradually dropped to 2k, then to 1.5k and finally back to the exact number of pages on my site.

(However, under URL parameters google reported that there are still around 2k+ replytocom urls on my site)

Result: My traffic gradually improved to 1500, 2000, 2500 and finally to 4500 visitors per day. I got really excited as I was on the road to recovery.

4. The traffic sustained for a few months but again started to decline. I noticed soft 404 errors in webmaster tools. Now google started to see mass 301 redirects of 3k+ replytocom urls as soft 404 errors. Traffic now dropped to around 2500 per day.

5. I finally decided to show 404 errors for all replytocom links so that they get dropped permanently and configured apache to serve 404 errors for all pages if they contain ?replytocom in their URL. After a month now I have 404 warnings in webmaster tools for all replytocom urls. As it was intentional I just ignored this.

But now the traffic seems to drop further from past few days. I am seriously worried. My question is should I issue a 410 response for intentionally removed replytocom URLs or lift 404 configuration all together. (BTW I am sure that 404 warnings are contributing to traffic drop even though google says that they don't affect traffic.)

I have messed up a lot with my site making too many changes after panda hit. Really fed up. Please help....

[n00b1]

Have you tried using URL parameters in WMT to make sure Google doesn't crawl them and making sure they return a 200 OK or equivalent like they would normally?

[sriki]

Yes I had configured google not to crawl those URLs. Even though they were all dropped from index they were still showing in the URL parameters page. So I changed the configuration from "do not crawl" to "let google bot decide" and added rules to my .htaccess file to return a 404 response for replytocom URLs. I thought this would make replytocom URLs disappear but they are still showing up.

Unfortunately, now around 600 crawl errors started to show up and traffic declined even further :(

[webmarketingxpert]

Hi,

As per what you have stated I think few steps may help:

First of all please ensure that your 404 page is informative and a justified page as per a good and effective 404 not found page is which also helps with sustaining the traffic around the site.

Check the crawl errors properly as to what exactly are they pointing to and if you are sure that you have resolved all then update your XML sitemap and resubmit it in WMT. I am sure you will see yourself heading towards reduction in the errors.

Let me know if it helps.

Thanks
Harsh

[kawen]

I also want to get rid of some links from google index, I have try 404, then I change to 410, I think 404 is same as 410 in Google eye, they create same error list in webmaster.

[aakk9999]

I think 404 is same as 410 in Google eye, they create same error list in webmaster

Pages that return 410 are removed from Google index faster.

And yes, both 410 and 404 show in "Not found" section of Google Webmaster Tools.

[sriki]

Even though Google says that 404 errors don't affect search rankings, I strongly suspect it really does. I think in the eyes of Google, a site with too many crawl errors/duplicate content is of low quality.

So, I have decided to lift the 404 errors and do a 301 redirect of all links with query strings like replytocom, showComment, utm_source etc. back to their canonical equivalents and see how this goes. I think this will get red of 404 errors quickly.

I am using Yoast Wordpress SEO plugin to redirect the replytocom links.

But my concern is that, will mass 301 redirection cause any serious problems again?

For example, if I have 100 unique posts and each one has 10-20 replytocom URLs indexed in google, will redirecting those 10-20 replytocom URLs to the same post (canonical URL) cause any problem?

Has anyone had such experience with replytom links in the past?

[JD_Toims]

3. I installed replytocom redirector plugin that 301 redirects all replytocom links to original URLs.

The traffic sustained for a few months but again started to decline. I noticed soft 404 errors in webmaster tools. Now google started to see mass 301 redirects of 3k+ replytocom urls as soft 404 errors.

So, I have decided to lift the 404 errors and do a 301 redirect of all links with query strings like replytocom, showComment, utm_source etc. back to their canonical equivalents and see how this goes.

So, you think the redirects caused an issue before and now not only are you going to remove the 404s, you're going to put more redirects in place. Not sure I see the logic there.

But my concern is that, will mass 301 redirection cause any serious problems again?

They were treated as soft 404 errors before and now you're installing more redirects. If you were a search engine and you treated the initial 301s as soft 404 errors, would you treat more redirects to the same locations as anything other than soft 404 errors? If you treated them as anything other than soft 404 errors, would you "score" them better or worse now that there are more?

Even though Google says that 404 errors don't affect search rankings, I strongly suspect it really does.

The only time I've seen anyone back up the assumption you're making is if there are *links* on their site to 404 pages, not 404 errors. It's been stated on a number of occasions 404 errors are not bad in-and-of-themsleves, and JohnMu has even gone so far as to say 404 errors, to him, indicate a properly functioning server -- Conversely, I have not see or heard of anyone at Google saying linking to 404 error pages would not be harmful in any way.

As far as duplicate content goes, Google "just picks a copy" to use when it's within a site and has a more complicated method when it's on two different sites. Where duplicate content [within a site] can become an issue is by "diluting" link weight when there are links [either internal or external] pointing to the non-canonical version of the page.

Personally, I think you're unlikely to solve your issues unless you start looking in other areas for the problems -- I'd start by making sure any *links* to 404 pages or redirected pages have been removed, then go from there.

[sriki]

So, you think the redirects caused an issue before and now not only are you going to remove the 404s, you're going to put more redirects in place. Not sure I see the logic there.

I am not sure about what exactly caused the soft 404 errors in the first place. Some reasons to support my confusion are:

1. Not all 301 redirects were reported as soft 404 errors. There were more than 2k 301 redirects but google reported only a few as soft 404 errors. How can this happen?

2. The recent version of Yoast wordpress seo plugin is also 301 redirecting replytocm links automatically to their canonical equivalents. This made me come to a conclusion that 301 redirects is not a bad idea.

After a few months after this, I deleted a few outdated posts on my blog.

However here there is a small catch which might have caused soft 404 errors:

1. When google crawled the replytocm URLs of non existing posts this is what used to happen.

domainname.com/deleted-post/?replytocom=123456

got 301 redirected to

domainname.com/deleted-post/

which shows 404 response.

So, instead of returning a 404 rsesponse in the first place itself, the server stripped the replytocom parameter and 301 redirected to a 404 (non existing page). This might have triggered the soft 404 error.

What do you say? Please correct me if my notion is wrong...

The only time I've seen anyone back up the assumption you're making is if there are *links* on their site to 404 pages, not 404 errors.

None of the pages on my site link to those replytocom links. So I decided to configure .htaccess to trigger a 404 response for all page requests containing replytocom parameter.

However, in the webmaster tools, under Linked from, I see google reporting that pages on my own site are linking to replytocm links. But these pages once did in the past (that is how google discovered the replytocom links) but not anymore now as I have removed all the replytocom links and cleaned the pages.

[sriki]

So, you think the redirects caused an issue before and now not only are you going to remove the 404s, you're going to put more redirects in place. Not sure I see the logic there.

I am not sure about what exactly caused the soft 404 errors in the first place. Some reasons to support my confusion are:

1. Not all 301 redirects were reported as soft 404 errors. There were more than 2k 301 redirects but google reported only a few as soft 404 errors. How can this happen?

2. The recent version of Yoast wordpress seo plugin is also 301 redirecting replytocm links automatically to their canonical equivalents. This made me come to a conclusion that 301 redirects is not a bad idea.

After a few months after this, I deleted a few outdated posts on my blog.

However here there is a small catch which might have caused soft 404 errors:

1. When google crawled the replytocm URLs of non existing posts this is what used to happen.

domainname.com/deleted-post/?replytocom=123456

got 301 redirected to

domainname.com/deleted-post/

which shows 404 response.

So, instead of returning a 404 rsesponse in the first place itself, the server stripped the replytocom parameter and 301 redirected to a 404 (non existing page). This might have triggered the soft 404 error.

What do you say? Please correct me if my notion is wrong...

The only time I've seen anyone back up the assumption you're making is if there are *links* on their site to 404 pages, not 404 errors.

None of the pages on my site link to those replytocom links. So I decided to configure .htaccess to trigger a 404 response for all page requests containing replytocom parameter.

However, in the webmaster tools, under Linked from, I see google reporting that pages on my own site are linking to replytocm links. But these pages once did in the past (that is how google discovered the replytocom links) but not anymore now as I have removed all the replytocom links and cleaned the pages.

[JD_Toims]

1. Not all 301 redirects were reported as soft 404 errors. There were more than 2k 301 redirects but google reported only a few as soft 404 errors. How can this happen?

A soft 404 is usually reported when a "larger" number of URLs are redirected to the same location or a custom 404 page is served but a 404 header is not set -- IDK if this is possible in your situation, but that's the "normal" reason.

I can see how redirecting to a 404 page could be treated that way for removed pages also.

What do you say? Please correct me if my notion is wrong...

Sounds very plausible/likely to me.

The thing about any of that is, all soft 404 means is: "We're going to treat this situation like the page is Not Found, even though that's not the status code we received." -- It's not a "penalty" or anything along those lines, all it means is if there would be link weight or other ranking factors passed through a redirect [or whatever] it won't be.

However, in the webmaster tools, under Linked from, I see google reporting that pages on my own site are linking to replytocm links.

To solve anything you absolutely have to be able to selectively ignore WMT. (I tend to completely ignore it, unless I have to answer a question about why something is reported the way it is in there on a site I'm working on -- By completely ignore I mean: some of my sites aren't even verified in WMT, because it's about as reliable and accurate as TBPR.)

What you *know* should always out-weigh what WMT reports.

If you know the links to the ? URLs are gone, then you can ignore anything WMT says about them, including reporting them as there.

If you know the pages you thought were an issue are either redirected, 404ed or 410ed, then you can ignore anything WMT says about them other than telling you what you already know.

In-my-opinion, it's a good time to decide how you're going to handle getting rid of the pages and links (sounds like it's already handled and either way you've tried should be fine) then move on and start looking at other areas that may be causing issues or could be improved, because there's nothing you can do to "fix" the inaccurate or out-of-date data provided in WMT.

[edited by: JD_Toims at 7:33 am (utc) on Oct 29, 2013]

[sriki]

To solve anything you absolutely have to be able to selectively ignore WMT.

But I cannot ignore 404 errors as they are causing a heavy drop in rankings and google traffic. There can be no other factor for drop in rankings other than sudden rise in 404 errors (afer making ?replytocom URLs show 404 response).

Now I am caught up with a double hammer:

1. If I mass 301 redirect those replytocom links, google reports it as soft 404 errors (as per opinion, but i am not sure) and drops my site ranking.

2. If I show 404 response for those links, again google drops the ranking.

3. If I leave them to show up a 200 response, google sees it as a duplicate content and again penalizes my site...

What to do... This is where i am stuck :(

[JD_Toims]

Google does not "penalize" for 404 errors or duplicate content within a site. Sorry, but I think you've gotten some bad info on the subject, because I have sites I work on with way more than your 3000 404 errors and it's not an issue. Like I said previously, Google just picks a version to use for content that's duplicated within a domain, they don't "penalize" your site for it.

There can be no other factor for drop in rankings other than sudden rise in 404 errors (afer making ?replytocom URLs show 404 response).

You mean, day-parting, query distance, location-based results, increase in ads on the SERPs, inbound link weight dissipation due to the cascading effect of networks being found and taken down, increase in popularity of a competing site and none of the 500+ updates a year Google makes could be the problem, it absolutely has to be the 404 errors on your site?

I think you should look in other areas and try to figure out what changed and what you need to do to gain traffic -- I also highly recommend building a user base from Google now while you have traffic, because there could easily be a day when you don't get any and there won't be anything you can do about it.