fishfinger - 12:37 pm on Jul 24, 2009 (gmt 0)

The directories could certainly be the culprit if they are the 'SEO friendly type' and identical anchors and descriptions were used.

You could have destroyed trust in your site with a bunch of crummy links, but the trust can come back with time. I've even seen cases where the site came back a bit stronger than before, but as I stopped using these sort of links some time ago I can't say if this is still the case. Google might not be so forgiving these days.