Looking back over our stats, I just discovered that we added ads to a large number of our pages the day AFTER Panda 1.0 (stroke of brilliance right there!) We were not affected by Panda 1.0. We were however affected by Panda 2.0. Although it was only one ad on the page, my theory is that these ads often appeared on pages with quite thin content (ie just one result). Maybe that thin content was considered ok without ads. But with an ad, not so good..
But ouch, if those ads we added were to blame, that's a seriously hefty price we've paid for adding Adsense. I've cleaned it up now so ads don't show unless there's a bunch of results and also ensured that the thin content won't be indexed. Here's hoping it helps.