Varun21, I don't think Penguin can be just about backlinks, because my backlink profile is weak rather than spammy, but I agree with Robert Charlton that in your case, it's likely to be the backlinks.
As for how Google would determine your intent... well, obviously they can't read minds, but I can imagine them feeling they have a pretty good "profile" of how an overly or wrongly optimized site appears. I.E., the backlinks look like this, the bounce rate is like that, etc. (I also suspect they would have lots of different profiles for differnt types of queries.) They might be ranking sites against such a profile, and if your site fits it, say, 85%, then they slap you.
Just to keep us from comparing apples to oranges, my site was not penalized according to Google. I lost about 80% of Google traffic, like you, but it was just due to changes in SERPS that happened on the day of Penguin. I think it's helpful to remember that Penguin is not a penalty, it's just an algorithm, and it affected a lot of people who did not have penalties or "unnatural link" notices.