Flash sites don't have to be large in file size – quite the opposite in fact: intelligent Flash coding and structuring of the Flash site will stream the site so that it delivers to the user what has already loaded, so that the user can interact with the "front part" of the site, whilst the rest of the site loads in the background.
So for example, if the entire site weighs in at, say, 100K, the user can access it after just 10K has loaded. If an equivalent 10-page HTML site weighs in at the same 100K (10K per page), you'd still have to wait for the 10K to load before you can access the page, but then when you click on "next page", you have to wait again, for the next 10K to load. In the Flash site example, after you've interacted with "page 1" (scene 1, in Flash terms), and you want to move on to "page 2", there's no waiting / load time, as it has already loaded in the background whilst you were looking at page 1.
Plus, when you consider Flash's use of vector-based graphics, it all adds up to small file sizes and fast load times.
That's what annoys me when people slag off Flash: they are slagging off the work of people who use Flash badly, and who have not created an efficient site. In the right hands, Flash is a very powerful tool (not just for "entertainment" sites, but for ANY kind of site, even corporatey, texty, informationy-based ones), and it's the bad Flash designer who should be blamed, not the application. Blame the workman, not the tool.
Trust me, I'm the Flash Lady! :)