I hesitate to even go here because feelings about Google run strong, but as I see it...
Honestly, I've never seen a site sink that didn't deserve it, and I'm talking about a couple that I've been responsible for that tanked. Quality sites that are regularly updated and have unique, expert information do as well as ever. Perhaps better because they aren't leaking traffic to spammers and scrapers.
A few years ago the Google results were so full of spam it was insane. I did some tests where I wrote articles of good quality, though hardly groundbreaking, and included some unique phrases that I could track. I would publish those pages in article directories and within 12 hours the first ten results were scraper sites with spun content often with completely nonsensical language (think "What's it like to SCUBA plunge in a classroom of fish?"). My original was quickly pushed off the front page.
Overwhelmingly in the last few years what has changed is Google has gotten way better at smelling the garbage. I've seen some of my garbage get smelled and literally drop off the grid. But it was garbage. I've heard of a few cases where the garbage smelling dial gets turned a little aggressively, but my garbage was garbage (not spam, but just non-expert content that in a print world would never get published; it was "good for the web" but garbage in any absolute sense). This was certainly true of the early Panda/Penguin rollouts that caused some collateral damage. That collateral damage got a lot of press, but the simple fact is before Panda and Penguin, I felt like Google was sinking in a cesspool and had become almost completely unusable.
Something I've been saying for years is that the end game of SEO is when search engines have the natural language and analytics abilities of a subject-area expert and the speed of a supercomputer. This is coming. Just recently IBM has decided to invest billions in Watson, the natural-language computer that defeated two Jeopardy champions a couple years ago. The current version of Watson is already 20 times faster than the version that won Jeopardy against two of the best humans to ever play the game.
To me, whoever can make the Watson technology work on a general task like indexing the web wins. Google will I'm sure spend billions to make sure that it's not IBM, though I don't think Google is yet especially strong in that field. But when you are trying to think of where things are headed, that's always the long-term target. We're a long ways off, but I think any technique that will withstand that standard (checkup by subject expert) will be very unlikely to become collateral damage in a Google update. It does happen, but it's far more rare than you might believe from all the accounts of webmasters who say their awesome site sunk for no reason (or, another version, their site which did will for so long suddenly and for no reason dropped).
So Google is miles and miles from a natural language search engine, but Google has expanded use of the knowledge graph way beyond calculating degrees of separation from Kevin Bacon. It has gotten way better at understanding all sorts of associations and connections and that means that it's harder to rank for content that is not useful. I think in your case, that's actually good news. If you have useful content, you have less competition from people who are aggressively buying crappy and obvious links to their mediocre content.
Why penalise someone for someone else's inbound links to your website?
Short answer: because Google's algorithm and infrastructure are not that
good. If the they were really good enough, they would simply recognize spam as spam and ignore it. The fact is, though, Google isn't actually good enough to sniff out all spam so Google relies on a two-pronged approach:
1) attempt to recognize spam and ignore it
2) bully and intimidate webmasters so they're too scared to try, thus reducing the overall stress on Google infrastructure.
So many people found we could get some pretty decent traffic by paying someone to do hundreds, even thousands of directory submissions. This was pure SEO BS and had no value to anyone, but it was cheap, formulaic and easy to hire out. The purpose was to game Google. When Penguin rolled out, Google was able to see that and one site I work on lost 90% of its traffic (they were doing these submissions before I ever came onboard).
On another site, which is a substantial company with namebrand, I got an impassioned query about why they suddenly lost all search traffic for one of their money pages. I didn't think it was anything I had done (backend stuff), but was a little worried I'd screwed something up on the server... until I saw that their inbound links for that page were all spam. They had hired an SEO company and stipulated that they didn't want any garbage links because they didn't want to sully the image. When I sent the link to the marketing director and he saw what the SEO company had done, he went ballistic. But it was too late. And realistically, that technique had brought them good traffic for three years. But there was nothing to do at that point but to own up, disavow the links and move on.
To get an idea of what sort of links I'm talking about that got this page dropped from the SERPS, imagine a knitting site where the commenter used the user name "Performance Corvette Mufflers" and then commented with "Wonderful article! I learn much interesting thing from your blog. Keep up the good work!" It was actually worse than that, but I don't even know how to make up something as bad as what the original SEO contractor had done.
Nobody would do stuff like that if not for weaknesses in the Google algo. So basically Google said "Okay, you want to screw with us? We're going to make it hurt if you get caught."
In my experience, 80-90% of the people who complain about unfair penalties from Google are simply not being open and honest with themselves about the quality of their sites. Demand Media might claim there was tons of great stuff on eHow, but mostly it was crap. And it was hit. Jason Calcanis may think Mahalo was an awesome site and Google was wrong to penalize it [2,3], but in 2010 Aaron Wall called him to the floor and criticized Matt Cutts for pimping spam when Cutts made positive comments about the site.
I'm mixing Penguin and Panda here since eHow and Mahalo were hit for thin content, not spammy linking, but just to say that these are some of the issues that Google was trying to solve while you were off running your real business instead of comment spamming sites totally unrelated to your business. I think that there have been a few times when spam began to choke Google to death and it was literally a matter of fix it or die. 2011-12 was one such time. Google results had gotten so crappy that I think Google was literally scared that their useless search results would create an opening of Bing could just get it right. So they got a little trigger happpy and hurt some good sites, but overwhelmingly they hurt the crap sites.