Argghhh... That is tough.
I don't know what to say. I have read somewhere on these forums that some people who received a minus 50 penalty moved their site to new domain and then 301 redirected the old site and the new site did not receive the penalty.
But you should research these forums for either OOP (over optimization penalty) or -50 penalty or such.
So I would agree with kellyman.