I recently updated my site to be much better structured and while I may be a small fish in a big pond (several hundred URL's) I think how URL's relate may have a lot to do with structure. I'll go over what I've done and whether it's relevant or not wouldn't mind constructive criticism though at the same time I think it may give some people insight. Honestly I think many site's have very poor URL structures which may be the owner/developer's inability to understand the content as a whole or simply the lack of perception that it's not organized. Often when working with clients I have to remind them that the way we look at a site is limited to us as others aren't building it.
I've structured my URL's the following way...
domain / section / area / page
We all know domains, duh. Depending on your site's content the way you organize content may vary. In example there may be a few "section-like" pages like about and contact with a huge very specific number of pages of say a product catalog.
Some "areas" of my site are direct pages themselves and I try to make "pages" not end with a trailing slash while "areas" that contain "pages" obviously contain a trailing slash; such "areas" contain an index with their own content to help human visitors better understand the "page" content with a bullet list of URL's.
Clean looking URL's with keywords seem to be the way to go...
I also use a hierarchy of links (#location) that clearly shows how URL's relate both to search engines as well as to human visitors.
Also I moved to a database which allowed me to automate a site index. With a mega-site you'd still be able to automate it with the URL limit on each site index page. I just don't see the point of having tons of content and no way to easily manage all of it. I think there's too much developer and way not enough designer in the mix. A developer makes something work sure, though you need a designer who understands development, humans and spiders to bridge between who is paying to have the site and the people building and maintaining the site. On top of that I think having a proper understanding of HTML (e.g. proper use of headers and keeping your content at the top of the source with non-content below your content) also goes a long way to help search engines understand what is content and how various pages relate. Also not having tons of excess code also helps with a greater content-to-code ratio. If a detailed page has a URL that has the same format as a lesser detailed though more mainstream page (e.g. about) then not enough work was made to make URL's humanly readable. Details still matter even at a large scale. I don't know if file extensions for content come in to play though I think offhand not having them would be beneficial, your keywords would weigh heavier on shorter URL's which may suggest the page is more on-topic.