"I would concentrate on the links reported in the Webmaster Tools on Google" "Do not worry about damaging other people, that does not happen" "Be aware of the site-wide disavow possibility, it will make your life easier" have been the key sentences in his reply. I am thankful for such a definite answer and thought I'd share it here.
Something that I think needs to be addressed more emphatically... when you remove links that have been discounted, you may not have enough remaining links to bring up your rankings to where they once were. The pressure to build these up artificially, IMO, will backfire, particularly because you will be under Google's special scrutiny.
Familiarize yourself with Historical Data patent...
It's clear that Google is looking at more than anchor text. Rate of link acquisition, covered in the patent, is IMO one such factor that webmasters and SEOs may be ignoring as they rebuild their links. I'd take it slowly, and I would start by cleaning up the site and adding new content to motivate new links naturally.
. Mod edit note: I've changed Best practices link above to the WebmasterWorld discussion of the sno.pe post noted above. Our discussion also includes link to the original article.
[edited by: Robert_Charlton at 9:23 pm (utc) on Jun 30, 2013]