Google speeding up wordpress with flush.
Look at WP minify to reduce calls if you have multiple scripts.
Look at self hosting CDN also so the browser will do more paralell downloading
Also get the yslow tool for firefox so you can start to see what is happening. It will tell you for example if your content is being gzipped.
I did a lot of research on WP speed and people quite often focus on what the server is doing and not how the browser is interacting with it. For whatever reason i found W3TC etc. slowed my site ( although im sure it might help smooth the bumps in a spike). I was able to move from an average 3 second load to 1.5 when i started to tweak based on what yslow was telling me. Oy yeah make sure your using browser caching where you can, even if your only setting the times at an hour it will help especially from the second page load and on.