Desert? Maybe not.
Visit less frequently? Absolutely.
If there was a one second delay for every page load I wouldn't visit, read or post anywhere near as much. (Don't give Brett any ideas...) There are days now when my visit ends when the site does the 'new post stall out' on me...
You might think 'consciously' the speed related to a second or two of load time doesn't matter that much, because it doesn't sound like it should, but IMO, behavior 'subconsciously' changes re a website that is slower to load. (I don't have any data to back the preceding statement up, but my gut tells me people say it doesn't, but their behavior changes when faced with the situation.)
I surf my sites all the time to see how they load and where the slowdowns are to try and eliminate them, because I'm on a fast connection and I know if I can see the page change I'm probably not quite as fast as I want to be for dial-up visitors.
I can't always get to where I can't tell if the page has reloaded without looking like here most of the time, but I'm usually close...
Anyway, I know from surfing my own sites, even the graphics that take a bit of 'extra time' to load (not the entire page) get to me after a while, so I try to find a way to eliminate the hesitations from the page displaying as much as possible.
There's one I work on and according to YSlow it's at about 79.4% for speed, etc. on the home page optimization / load time (whatever they base that percentage on... I don't have it installed right now or I'd be more precise).
Anyway, I also know that's a script telling me about it, and I've opened it on dial-up a couple times and the real-time 'request to display' on 56K is less than two seconds all the time and usually under 1 sec. (If you have the graphics cached... They're about 1.5-1.75 secs or so from page-open to load. So without the graphics cached: under 2 is the norm and if you have 'em in the cache, under a second.) It's got a graphic header about the size of the header here, with a graphic footer about the size of the crumb line at the bottom of the page and a banner... Most of the rest of the page is designed using CSS rather than images, but it's a dynamic directory, so this includes all DB connections, page parsing, mod_rewrite, and everything else that makes a php driven site with friendly urls work.
If you want to have some fun and see my point some time, back up one of your homepages, upload it to a test server and use JS or a server-side lang to slow it down, then refresh it a few times and sit there and watch it load... Empty your cache, refresh some more, empty some more, then when you get bored, go back to your 'regular speed' home page and do the same thing. (This is what happens when someone visits that page frequently and doesn't have a cache of it... You're just compressing their visits into a shorter time span to see what they go through to open and view the page.)
When you're done with that comparison, take the test page back to 'regular speed' and start optimizing for speed. Pay careful attention to which graphics load first, which delay, what you can see before the graphics are loaded, where the 'bottle-necks' might be and start playing with load order, text length (within the source code), etc. to see how much faster you can make it actually appear in the browser (using the suggestions from some of the tools is cool, but IMO the only way to really tell the difference and impact the difference may have is to sit and watch it load over and over.)
When you've got it as fast as you can make it, go back and surf your home page again... My guess is you'll optimize your entire site and think speed is highly important to visitors, even if they tell you it isn't.
EDITED: Added clarification to some points.