I had one site I got out of that type of penalty by looking at 'where' the backlinks were from. Many were well beyond the niche's neighborhood and quite a few were not in the adjacent related niches. The person assigned the linking task only used 2 anchor text keyword variations and never the URL.
After removing ALL of the links that were well beyond the niche's neighborhood (totally unrelated) and most of the spammy looking 'adjacent neighborhood' links did the site bounced back.
The person doing this linking never triggered a un-natural link acquisition response in WMT too. So I didn't get wind of it until the customer made a comment about a 'grey bar' PR Toolbar rating. It took quite a few weeks for the site to return to PR3 from a 'grey bar' PR0 rank.