Well, the gist of it is, you lose a LOT of page rank / link juice when you remove / disavow links to escape Penguin. By it's nature, it was designed to detect links that were giving you more juice than you deserved. So by all intents and purposes, as I understand it, your site didn't have a legitimate reason to rank that well in the first place.
If I site had 100 backlinks all giving page rank and then 50 of them had to be removed or disavowed, well, you are only going to get half the page rank you had before, right?
So you are going to have to get LOTS of legitimate links to make up for the 50 you lost.
So yes, it looks like people will have to think twice, but the other options are 1) give it up entirely, or 2) move to a new domain, or 3) do some more blackhat / spammy stuff and hope they don't get penguinized again.