Okay, since the Panda update on the 29th, I've seen a steady rise in traffic. It was hard to tell at first whether this was related to Penguin, since I've made a lot of other changes which took the rankings even LOWER than Penguin did.
But today I'm seeing a lot of old inbound referrals that I haven't seen much in months or even years. I'm wondering if those sites have just recovered from Panda, and they're getting more traffic to send me, and also their inbounds might now have more value for my site.
I think this supports what you've been saying, Robert_Charlton, about being affected by backlinks even if none of them are particularly "spammy." If enough of them have been affected even mildly by Penguin OR Panda, the cumulative effects could impact your site deeply without you having done anything "spammy" or even SEOy.