hi coastal, i cannot really tell but i suppose the numbers of simultaneous users on your server and the speed of your server in general will influence the time it takes to load a site more than the protocol that is used since httpsecure just means that a certain port (443) is used plus some additional checks are performed... well i'm not too sure about this, never tried it myself.
about the loading of cached formerly http delivered sites into the browser when those sites now are delivered via https (if i got that right?): this might be a concern of the cache's config, however, you can influence the caching in many ways: [psychotekk.de...]
When I access my secure page from within my site, there is a noticeable slower load time. I think you are correct PsychoTekk in that the browser has call everything again via https - I know I have to code all images that way.