I've been wanting to get to this for a while, as johnser's description rings some bells. He wrote...
A well-deserved manual penalty was applied c 5 years ago. Think "minus 30 penalty" from c 2006.
That manual penalty is somehow still in force on a narrow range of terms like "Red Widget, Cheap Red Widget" etc despite everything possible done SEO-wise to clean up the mess and build a pure link profile. Nothing has worked.
I remember encountering pretty much the same pattern on several sites, one in particular, around 2006, where the target terms were "Cheap Widgets", "Cheap Red Widgets," and "Red Widgets". What Google was doing in those days, as I remember, was cycling through various keywords, doing comparison scans, and in this particular series they went first to the most competivive (and for us, most desirable "Cheap Widgets" which was almost a holy grail for SEO)... then "Red Widgets"... and then "Cheap Red Widgets". Sometimes we survived the winnowing... sometimes we dropped. We wanted all three phrases, of course, and via a combination of tweaking content, repurposing some pages, and focusing some internal links, we had the two easier phrases in the top 2 or 3, and then "Cheap Widgets" on and off the first page, I think as high as #4 or #7 for a while.
This was a low budget site, and we were competing against very well-funded exact match domains. The owner was for building more exact-match anchor text links into the site... and I was opposed to the exact matches, to the speed he wanted to build the links, and to several links that obviously weren't natural.
I bring these pressures up because the more we fiddled, the more volatile some of the rankings became. I had my hunches, but it wasn't until six years later that they were confirmed. Google, IMO, had been setting traps to gauge user intent. The patent was widely discussed in this thread, which I recommend... Google's Rank Modifying Patent for Spam Detection Aug 18, 2012
When Google suspected manipulation, the page expected to rise might move down a page or two, effectively to check intent. I've seen the pattern enough now that I strongly believe the patent is in use. As I remember, the patent even suggests at one point that the motivation for manipulation was owner pressure on the SEO. I've assumed this was a Matt Cutts contribution to the patent.
In our situation, for six months, the owner took the site over to build low budget links offshore; but they were incredibly obvious and tanked the desired terms. The client eventually cashed in on what was left of the site. What I've observed since then is essentially speculation and hearsay, but nothing was done with the site, and certain penalties apparently wore off in six months, and others in a year... and "Cheap Widgets" page-one rankings never came back.
I mention all this to suggest that the problem you've been experiencing may not be what you haven't been doing, but rather what you have been doing. I would think that the optimization efforts focused on a few closely related pages must be obvious to Google, and because of earlier spammy attempts that you describe, Google may be assuming that this is a continuation of the manipulation of what had been done before. .