goodbyedee - 11:25 am on Aug 16, 2010 (gmt 0)
How old is your site? For how long have your site been at 2-3? For how long are you seeing your site going down?
I don't believe that the links hurt you. Maybe they didn't do any good, but they didn't do harm either.
You should definitely be patient as indexing of changes has become very slow lately.
In general my advice would be to improve you content as much as possible, make sure all on site optimization is done (Titles, Headers, alt, etc) and start getting inbound links from related sites.