That certainly does sound like a penalty - but it's the reverse of what many people suffer. It's more common to see rankings fall for the main keyword phrase.
This does not sound like a very diverse backlink profile. I'm not saying that is definitely the cause of the lost rankings, but I'd suggest taking a close look. For example, is the anchor text natural - that is, not a high percentage of "keyword1 keyword2"? Is there any pattern you can see between lost keyword rankings and those keywords in anchor text?
Also, are there at least some freely given "editorially placed" links - backlinks that are not obviously arranged and coordinated by you?