Yesterday GA reported an average page load time of 4.14 seconds, with times ranging from 2.08 to 7.58 seconds.
I had to clear Hyper Cache's entire page cache several times yesterday due to some global structural changes I made to the site, so I'm thinking that the average load time will come down even more when I check it tomorrow.
Things are looking better. However, when I'm logged-in to the site (which means I'm not getting cached pages) I still experience the same thing that edacsac describes. Saving a post update takes anywhere from 5 to 45 seconds.