robzilla - 10:42 pm on May 19, 2011 (gmt 0)

If you have your own dedicated server, or even a VPS, and you're not using up the available resources, there's probably no need to upgrade. If you're on shared hosting and you can reasonably afford to scale up, then you should certainly consider doing so, even if only to be able to exercise more control over how you serve your content.

And if possible, make sure your server is geographically located near as many of your visitors as you can manage.

When it comes to your site's performance, just ignore Google. The idea of upgrading a server because it may or may not affect search rankings reminds me of the webmasters who, several years back, extended their domain registrations by several years simply because the thought circulated that Google evaluates a domain's age or, by extension I suppose, the time it's registered for. It's simply not where your priorities should be. Focus on the things your visitors will notice.