I had a situation recently where a new client came to me after doing some of their own shoddy SEO. They had very few links to begin with... maybe a total of 20 or so unique referring domains. She had done some on page keyword stuffing (which was quickly and easily changed) but had some very obvious anchor text backlinks. Her site fell to past page 24 after Penguin. I had her contact the sites with spammy links and they were eventually removed. She now ranks on page 6 for her main phrase. This appears to be an algorithmic improvement as it happened around the 24th of May and there was never any notice from G indicating a manual penalty. Personally, I'm very encouraged by this... seems to indicate that removing enough crappy backlinks can automatically free you from penalty box.