Welcome to WebmasterWorld!
Try not using plugins - or at least using less. :) Seriously.
To add to the thread phranque pointed you to, keep your database trim and lean. Despite caching, sooner or later the db needs to be queried to get the latest content. Optimize it, get rid of post revisions, and look for errors.
Also, reduce the http calls - use one style sheet instead of two or three. Use a sprite instead of two images to create a rollover effect. Each time the page has to make an http call to a resource or each time it queries the database, is another few moments the user has to wait.