in fact I haven't done anything for over a year with the site
In our very competitive vertical, that seems to be a common denominator with Panda. Sure there are usually big brands are sitting pretty in the top several results, which isn't surprising. What is surprising is that the sites that were hit years ago and completely neglected (no new links, content dated 2008,2009) are back in mix - and there are several commanding very high results for very competitive keywords.
One common denominator of these sites is doorway pages. I doubt Google is that dumb, but they're working like a charm for multiple sites. Yuck.
It's almost like Google is choosing the lesser of all the evils - if you just stayed on the sidelines for the past 2-3 years and did nothing you're being rewarded for it. If you played the SEO game you're dead meat unless you have a serious brand.
Bottom line is that the results are filled with so much junk - I have absolutely no doubt that MAJOR adjustments will be made in the organic results for our vertical. It's easy to get caught up in all the doomsday scenario's posted around the web, and if Panda improved the serps in your vertical then I'd be worried - but if not then in time it will have to be fixed.