I had a similar thing happen to me earlier this year.
Very diverse sites, about a dozen. All banned. Google only. The thing they had in common was adsense, or had adsense in the past. Little or no interlinking.
I'm very paranoid now, and much of the content I'm adding to the web is on third party sites. I think it's good content and it tends to rank fine.