I use a dedicated server for that reason - speed. We have fairly consistent load times in Google Webmaster Tools. We have 2 gig of memory available and typically only have 1 gig active and 1 gig inactive on a FreeBSD server. Loads are almost always under 0.1. Our entire website and images including discussion forum is about 150 mb.
I see almost instant response when grepping from a file that is updated just once a night. My guess is that our daily active files are in memory and there is no disk access time.
My guess is that on a shared server, besides server load to contend with, your files may not be in active memory and need to be loaded from the disc.
[this was added after the posting above was made as I was writing - I suspect that a VPS - virtual private server - will have the same memory problems if your calls are not constant - the server will push your files out of memory and have to make the disk call again]
If you are making money, I would recommend going to a dedicated server to eliminate any problems.
My load times dropped from 4-5 sec to 1.2-1.2 seconds with no apparent change to our users.