Also make sure you have the right wordpress plugins and theme. They can help speed things up.
What are the right plugins?
There are too many good options to name. Make sure your plugins are not redundant (like generating duplicate meta tags). Also look at the caching plugins. As for themes look at how much code is needed to generate your appearance, you can sometimes trim a bunch of code (making the pages faster) and still have a good look.