I read (some time ago) that you have fifteen seconds to make a good impression; I would not be at all surprised to learn that this timeframe has been shortened by now.
And besides, that's fifteen seconds in which the user will decide to stay or leave. Your page needs to be significantly loaded before that limit in order for you to have much chance of the user's decision being in your favor.
Of course, fifteen seconds on my sister's twitchy dial-up connection was quite a different thing from fifteen seconds on my cable-internet connection. Have you done any research on the sorts of connection-speed most of your users are likely to have? This may have an impact on how "large" you decide to make your pages.
Keep in mind, though, that even graphics-heavy pages should be designed to load fairly quickly. What sorts of pages, content, and/or page size do you have in mind? Do you "need" to have large file sizes?