With Penguin, we lost 25% of the traffic to our reference site. We have done no link building, sharing, or swapping of any kind, and we can say this with no qualifications. 99% our work goes into the site itself, and the 1% of work that is offsite consists of managing our social networking profiles.
We do have spammy links to our site--including one domain with over 220,000 links to one page on our site (we requested that these be removed and that webmaster complied, but only about a week ago, and WMT still shows the links)--but we had nothing to do with them. Honestly, we never paid much attention to these links when we saw them in WMT because we figured Google was smart enough to figure out we had nothing to do with them.
So Penguin is not all about link-building, as we have done nothing of the sort. One potential issue we've considered is that a large portion of our backlinks are from comment sections and forums. These tend to be very diverse in nature, with links coming from news sites, sports forums, video game forums, livejournal discussions, Reddit threads, and on an on. I won't describe the subject of our site here, but I'll just say that people often share our content to settle bets with each other or to prove one another wrong, and our pages are very shareable for those purposes. We've considered the possibility that our content is actually TOO shareable, and that this is triggering something at Google. It's all very frustrating.