I made some htaccess file changes and things where better I thought, but it's a random occurrence. Just now I loaded up my site from work - over 5 minutes to load. Switched to a different post within my page: 3 minutes to load up. Then all of a sudden it was back to 2-5 seconds. But this happens everyday. I generally won't wait 1 minute for a page to load, even if the odds are 1 in 10 times it will be slow.
I think I'm going to dump wordpress. Any ideas on transferring the setup to another CMS? My only problem will be my photo gallery plugin; they don't make one for other cms's, and the work to start over will be tremendous.