Well I have some news to report that may help some people.
About 10 days ago I removed a single large block of adsense from one of my sites. It was one of the large square blocks, placed atop the content, front and center. The site has been struggling quite a bit for some time, though I didn't know why.
It's a quality site and has had great rankings in the past. I haven't changed anything on the site in at least 6 months. 10 days after removing the ads, I got all my previous rankings back. My traffic has tripled ( from 500 uniques per day to 1500). It's conclusive to me that ads can definitely hurt you, and very badly.