In my case, no, I'm not gun-to-my-head certain that those links are what caused my trouble - because Google doesn't tell us why we are penalized. It's all speculation, theories, and guesswork. However, the penalty hit me on April 24, and that was the day that Penguin was rolled out, so the most likely explanation is that Google thinks I have spammy links.
I know for sure that I've never bought, sold, or traded links, so it's nothing I did in that arena.
Of course, there are plenty of other theories. Maybe sites that link to me were knocked out of the index because they were linked-to by spam. Maybe I'm being penalized for the blogroll-type link from a subsite of proboards.com (which I never asked for, but which they agreed to remove - not yet reflected in WMT). Maybe, coincidentally, Google decided to Panda-ize me on the Penguin day. Maybe there was another change to the algorithm that rolled out on April 24 that affected me. Lots of "maybes", but no way to know for sure.