Panda Recovery - Specific issues

[Pandacide]

Hi,
Our site got affected by Panda between Jun 2011 and September 2011 when we lost 2/3rd of our traffic. Recovery efforts began only by Jan 2012 after I came in.

Penalty:
Our site is a Q&A site which publishes original threads as individual pages with related Q&A content. Google is quite happy with this. The problem was in "search results" pages that had long-tailed URLs and questions from our DB based on those long-tailed words. In these pages, we showed around 15 questions linked to their individual question pages. There were "quite a bit" of them, more than the number of actual Q&As. Panda (unsurprisingly) soundly thrashed them.

Recovery strategy:
We have got 90% of our "search results" pages removed (404'd/410'd) in stages, starting January and the results generally, have been encouraging. Individual thread pages gained more traffic than that lost through "search page" removals. This was till Aug 26. Oh. We also publish thread pages daily as users post them and gain traffic every 15 days or so because of that.

Problem:
In the 2nd week of August, the index of the "search results" pages started increasing and went up 4-fold though we did not add any pages. In the next week, the traffic in the "search results" pages went up by 15%. However, in the following weeks, all pages started receiving less traffic. The index remains around the 4-fold range. We checked (ad infinitum) to see if any extra "search results" pages have inadvertently been published. Looks water-tight. The only thing is that we increased the crawling rate around 20th Aug.

Some supporting information:
1. Index is close to a million (with the bloated "search results" index that makes 50% of the site index)
2. We track index sizes of each of our subfolders, especially to see the increases in index sizes of our daily-published content, and also because GOOG's "site:" search with a date qualifier usually returns wrong results


Questions:
1. Why would the index go up so much so suddenly? From where can it get so many pages suddenly?
2. Could the (slightly abnormal) traffic increase in the week after the index size increase and its subsequent reduction be a result of this index fluctuation?

Thanks for your time
Pandacide

[tedster]

Our site got affected by Panda between Jun 2011 and September 2011

Your problems may not be related to Panda at all. Panda was run four times in that time period - Jun 16, Jul 23, Aug 12, Sep 27. If your drop did not happen on one of those dates, then it's not Panda. And trying to solve a Panda problem that really isn't Panda can be a frustrating dead end.

1. Why would the index go up so much so suddenly? From where can it get so many pages suddenly?

This is exactly the question you need to dig into. I've coped with more than a few problems like this, and the answers came in many different forms - various "sort" options, query strings getting added to URLs by various kinds of tracking programs, other query string URLs being generated accidentally or maliciously by other websites... on and on it goes.

Once you can find the pattern, you can take steps to disallow it from crawling and indexing. But you need to know what it is first.

Even on a huge site, you can often find examples of URLs that you didn't intend to be indexed by using the site: operator with a directory - e.g. site:example.com/directoryname/

2. Could the (slightly abnormal) traffic increase in the week after the index size increase and its subsequent reduction be a result of this index fluctuation?

It is possible - hard to say for sure. At some point after piling up spurious URLs, there would be automated safeguards that kick in,

This sounds like a situation where WebmasterTools might give you valuable insight. Have you set up an account there?

[Pandacide]

Thanks a lot for your reply, Tedster.

Firstly, I am sure it is a Panda issue. When I checked our traffic drops with Panda updates, I saw clinching evidence. Also, I know what the site was trying to do then and it definitely attracted a penalty from the Panda update. So, let's park that aside (safely :) )

On your points regarding "site:" searches and WMT:

WMT shows a constant decrease in the index size (consistent with our deletions/404s) though "site:" search has started throwing up huge numbers in search results since around 20th Aug. And in tune with the increase in index size (through "site:" searches) is the drop in traffic. That is something that has me worried.

I have correlations but no causality. I wanted to know if you or any one in this forum has experienced something like this before.

Thanks again
Pandacide

[Pandacide]

Well. Since last Wed, we are getting back the lost traffic and may even overshoot. Don't know if anything changed in the mean time.

Pandacide

[edited by: Robert_Charlton at 6:08 pm (utc) on Sep 22, 2012]