In one of the best conferences I ever attended I learned this:
Study between Amazon.com and About.com *4-5 years ago
100s of users testing the sites. Everybody said Amazon was 3-4 times faster and About.com, after tape reviews it was revealed that About.com was loading 5-6 faster than Amazon.
Difference, users at Amazon were abble to accomplish their goal. About.com was a difficult site to navigate.
Think about the goal of your users when they come to your site, make it easy for them to accomplish this goal.
Sometimes you will do things that you personally don't like but your visitors do..
All comes down to Testing, Testing, Testing...