A few months ago, I started to notice we were dropping for some fairly decent keywords we have held for a long time. I simply wrote it off as part of riding the G Train in terms of SEO.
This morning I logged into Webmaster Tools (which I have not done for some time) and noticed under "What Google Sees" => "In external links to your site" that instead of all the great anchors for inbound links we previously held, the list is dominated by what apppears to be spammy or unrelated anchors.
technically they will not penalize you for inbound links... they just drop the link influence... so where are the others or have they been dropped and the crappy ones are left there for now.. soon to be dropped too
To answer your question I don't think anything can be done at this point except trying to get more valuable links to your site.
Google is very discreet about that, they won't openly say that there is a major problem. There are so many pages over the internet that they still find a way to bring relevant results even if some sites get sabotaged.
I am in the same situation, I dont know if it's sabotage or no luck but it definetely screwed a site pretty badly. Links appeared overnight from irrelevant sites, mostly on similar IP's. Good luck.