TheMadScientist - 7:44 pm on Nov 27, 2012 (gmt 0)
By the way, Google continues to say that the 404 errors don't really cause SERP penalty and I've been told to look elsewhere. However, it's a fact that the rise of 404s and traffic drop are in sync.
That's true, and welcome to WebmasterWorld ... Here's your first 'gotta read exactly sometimes' note ;)
You're correct, 404 errors are not the issue, because if they were a competitor linking to your site and creating 404 errors could tank you, and if 404 errors were an issue removing thin pages that should not have been on a site in the first place could tank you BUT I didn't say 404 errors were the issue, I said it was the broken links on your site. (IOW: Broken links TO your site and broken links ON your site are two totally different things.)
IOW: It was not the 404 errors on your site I was talking about, but rather the links that were pointing to 404 error pages. (There's a huge difference).
2. We installed the tag auto-link plugin because we loved the concept that's been on Engadget for long. It helps users find more posts related to any topic. I disabled it because I was told that it could lead to several 'similar pages' on our website.
If it's really good for visitors I would try and find the fix for the similar pages (like making sure a rel=canonical is on every version of the page(s) and points to the main page if a 301 is not possible) and so what if your not selected page count goes up because the duplicates are not included?
TMS Pauses & Wishes Google would explain their sh*t to people better so we didn't have so much *bleeping* confusion about things, because it's absolutely ridiculous to have someone thinking they need to tone down visitor experience to manage a stupid number reported in WMT assuming the management of the number will make Google 'happier' than a better visitor experience would ... There should be rules and required (detailed) explanations for 'new toy additions' to WMT.
3. I'm worried about 'not selected' URLs because I've been told by the experts(?) that the trend in the increase of 'Not Selected' that we've on our site is quite the same as that of those website penalized by Google. Frankly speaking, I've fixed all the problems on the site and back to the default setup that worked fine. The 'Not Selected' is the only 'issue' which is yet to be addressed, if it's an issue at all. I don't know what else I should be fixing to get all the Google love back
Again, the detail in the wording makes a huge difference.
A penalized or filtered site will generally have a decrease in the number of indexed (selected) URLs, but we don't see that from the chart you provided, what we see is only an increase in the number of URLs Google has decided to not include in the index.
So, what we can see is:
Google found a bunch of new URLs.
We know they found those URLs to be 404 errors.
We know the 404 errors specifically are not the issue.
We know the URLs should not be included in the index (selected), so them not being included is fine and we don't need to worry about that. (It would actually be worse if they were included, because they would likely generate some 'negative signals' when visitors clicked and then went straight back to the results only to immediately click again, or worse, block you before they immediately click again.)
We know 'user experience' is a ranking factor.
We know broken links are a bad user experience.
We know (unfortunately) the links to the 404 pages Google found were on your site.
We know your traffic started dropping around the time the 404 links (which generated the 404 errors and the increase in not selected pages) started showing up on your site due to a malfunctioning script.
We can reasonably conclude:
The Links to the 404 pages decreased the 'user experience score' (for lack of a better phrase) on your site and your rankings were decreased accordingly.
We know you already fixed the linking issue, which we can reasonably concluded caused the issue.
We also know trust is a ranking factor and it's reasonable to conclude having a large number of broken links (not 404 error pages, but the links to the 404 error pages) on a site, which can seriously decrease the quality of a user experience, could have an impact on the 'trust' awarded to a site.
My personal advice is:
Forget about the not selected pages, because managing the number is not likely to help you, go back to building a great visitor experience, get some more quality inbound links and let Google work the rankings out.
I would guess since the 404 links were on your site for a period of time it will take more time to recover than it would if they were only there for a couple days, so I'd personally go back-to-the-basics of traffic/ranking building (quality site, quality inbound links, great visitor experience) and let things work their way through G's system for two or three months, then give it another look if things hadn't started recovering.