I read an article that I cant link to because I'm a new username here that said the exact same thing happened to the guy. In the comments was another person who the exact same thing happened to and after he removed all the adsense his rankings in google came back.
So I want to see what they will do, if the rankings come back maybe I'll try only 1/4th as much ads or just ads on the main page and see if the penalty comes back. Most of the sites are making such small amounts now with 0 google traffic that its worth a try.
I'm going to start looking around for ways to make money on websites without adsense and starting building new sites so that I'm less dependant on adsense.