bhartzer - 10:01 pm on Jan 20, 2014 (gmt 0)
RajaAdnan, what you'll need to do is go through all of the site's backlinks and determine if the link is a good link (natural) or a bad on (unnatural).
I would then notify site owners to get those links removed. Disavow the links that you couldn't get removed. Then, you'll need to continue to build good quality new links to the site.
If you need to make changes on the site, make those changes for the users and usability, not because you might have some sort of penalty.