Hey, it's generally pretty easy to figure out what problems you have as long as you have Analytics installed and can review the timeline.
I have picked up a client that who has, it would seem, pretty much every penalty going. They have a manual penalty from before Penguin and Panda and subsequent analysis shows further drops on all the relevant panda and penguin dates.
We have been working to resolve the issue(s) and have made some improvements by removing panda targeted pages and removing bunches of backlinks from some of their own network of sites but as of yet, whilst things are better, we have not been able to resolve the manual penalty and it would seem to get out from under that rock, there is more to do.
So, point being, don't think this is a one penalty gig, you could have problems with a manual penalty, panda and penguin and all of them are contributing.
There are some good resources now for diagnosing specific issues and you can likely resolve some, if not all of your panda problems without too much trouble. Penguin stuff depends on your link building and how much control you have over the link profile but seeing as you have a manual penalty as well then.... I am guessing you may have further problems.
There are some fairly solid resources out there now on how to diagnose and at least determine how possible it is to recover but if you are in big trouble, it may be easier to kill this domain and move the site to a new address.
Ultimately, it's impossible to advise without a link and some analysis, but there are ways to get an idea of how much work is involved and whether you are best to start fresh or persist with your clean up efforts.
Hope this helps!
[edited by: tedster at 1:39 pm (utc) on Sep 28, 2012]