It is also an issue of the kind of banners we are seeing today today. The banners I am seeing are for things like teeth whitening, education scams, acia berries, date sites, etc.
As someone mentioned up topic, what matters is the the brand impression you make. I am getting very low band impression on banner ads these days.
and if you sell ads to a low quality advertiser, it lowers the trust folks would have in your site.
I have seen one site that has maintained very high click through, but the owner only takes ad for products he trust, the ads have the name of the company in the ad and the product is relevant to his users.
I think the google adsense ad is going to murder click through, as lots of the google ad sense ads are exactly wrong for a site. On a couple occasions, I have seen google adsense ads that were offensive to visitors on a site I no longer go to.