I removed a lot of links that I assumed were harming me and I have dropped further.
I suspect backlinks are involved, which might take some time for Google to reindex. I also suspect a recursive element: Penguin penalises your position by a factor. The next time it runs, it again penalises your position by that factor if whatever caused the penalty is still there.
One page of mine - no spam, no solicited backlinks, not commercial, written solely for content - went from page 1 to page 3 to page 5 to (Thursday) page 39, without any changes to the page that would affect position.
If general page 1 results are anything to go by, Google's self-congratulation is delusional.