The thing about Penguin, as I understand it, is that it looks for artificial footprints, then smacks the site in question. The inherent value is basically irrelevant (apart from possibly raising the "score" that the demotion kicks in at).
Thus, if you are able to "fix" your backlink profile to make it less prone to Penguin, you are just confirming that you are a dirty spammer in the first place. Spammers don't get rehabilitated.
Panda, like claaarky says, is about user experience. I don't personally think it uses actual metrics in the way claaarky does, but I agree that changing the UX changes the Panda action. Sites can and do improve, and Google is willing to reflect that improvement.
As such, I would be far more worried about a Penguin hit than a Panda one.