I made a small title edit to clarify that we're not talking about a traditional "Google Whack."
Yes, you've lost rankings. The question is whether that's a change in the algo, or an algorithmic penalty, or a manual penalty. From your description, it sounds like an algorithmic penalty to me.
I'd stop building those junk backlinks with keyword anchor text and work on attracting strong natural links.