yosmc - 12:13 pm on Mar 23, 2013 (gmt 0)
I run a special-interest widget directory that is older than Google itself. It is useful to human visitors (in fact it's even been featured in offline media), it's been ranking well since the old Google days, and has never suffered any major drops from any of Googles updates or algo changes.
Over five years ago, I changed my site's back engine. The change was invisible to visitors, but internally, I installed a link management system that would help me manage and update my directory more efficiently. The script defaulted to a /links directory, so instead I tweaked the code so it would output to my traditional directory structure (along the lines of blue_widgets.php, red_widgets.php, etc.)
Middle of last year, Google suddenly started sending me warning mails via my Webmaster Tools account, telling me about "possible outages" and that "Googlebot can't access the site". I looked into those, and noticed that Google had found (and decided to spider) the default /links directory, even though it isn't actually linked anywhere on my site. Thinking that Google had no business crawling that directory in the first place, I simply ignored those messages.
Obviously, I shouldn't have: Over the course of 5 months and accompanied by a total of 30 warning messages, Google eventually started slamming my site, moving it down from page one to page 80 and beyond.
When I finally figured out the devastating scope of what was going on, I blocked the "links" directory via robots.txt and filed a removal request for all the "pages" that Google took from the /links directory and which it shouldn't have crawled in the first place.
As a result, the number of my indexed pages dropped from "several thousand" to "several dozen". This is really how it should be - like I said, this is a speciality directory, and there are only so many links in it. And: this is really how it always was, at least before Googlebot started snooping around in a directory it was never invited to.
"Index Status" in my Google Webmaster account, however, now looks like I have essentially taken my site down - if you don't know the full story, it seems like I removed 98% of my site, and put it into hibernation mode. And that seems to be exactly what Googles algos are concluding: even though my site is perfectly healthy, Google is still treating it like a half-dead zombie.
Can anyone recommend a path out of this nightmare, or should I just go and shoot myself? :p