have you tried checking how things look in a waterfall chart? you can do this using built-in developer tools in many browsers such as chrome and safari. this should tell you which component is taking a long time and you'll see how much is latency and how much is load time.
i would also try things like nslookup and traceroute to make sure it's not a DNS resolution problem or network problem between you and the server.
I've seen this before but it's quite often one of the following:
A client site had an issue similar to this and I checked all of the above to no avail. For several months I banged my head but couldn't find anything. The the next WP upgrade came out and it also upgraded the the database. Voila. What was taking 20+ seconds to load came down to 4. I'm still not sure what it was but we're happy it's better now.
The "stock" rewrite reads the entire file system twice
this isn't always the case. as far as i know, most modern OS, including linux, bsd, OS X, windows & IIS use a "filesystem buffer cache" so files that are often and/or recently accessed will be reaccessible almost immediately without a physical disk seek.