Sand - 1:34 pm on May 15, 2012 (gmt 0)
A good friend of mine was hit, and he's done absolutely zero linkbuilding. He doesn't have a lot of links mind you, but all the ones he does have are natural editorial links.
In his case, he was too aggressive with on-site factors (for instance, some of this title tags were keyword1 | keyword2 | keyword3, and his internal links all had heavily optimized anchor text). I obviously can't say for sure that that's what hurt him, but I've pretty much ruled out external factors being to blame.
I'm sure you can still get hurt by links, but I'd recommend that you don't focus all of your efforts there.