I don't really have a lot to add honestly.
Again, I don't want to present myself as a server ninja. I'm not.
That said, you can always make good use of more memory :-)
I would say that you need to find out where your bottlenecks are though, principally
If your system is bogging, you might want to start by profiling Wordpress, but you may not have control over this. If people absolutely have to have a given feature, telling them it burns a lot of horsepower may not change anything.
Quick example - I had a site that was bogging down after some changes. One change I made was using PHP to grab the image size so I could inject proper height and width attributes into the HTML for the IMG tag. When I profiled the site it turned out that about 80% of the time was spent grabbing these dimensions on a gallery page. Simple fix and now, in the age of responsive sites, it's better to avoid coding dimensions into your image tags anyway.
But assuming that you can't do anything at the software end, you can always shift things around.
If you're CPU/DB constrained, for example, you can put a reverse proxy (Varnish, Squid, etc) on the front end and do a lot more caching.
If you're I/O constrained, you can load things into memory with Memcache so you get a lot fewer disk hits. I'm not sure how big your DB is, but with an extra 4GB, you might be able to just load your whole MySQL database in memory with the understanding that a system crash between writes to disk would cause data loss (a reasonable tradeoff for a pure content site, but not acceptable in other contexts).
And then, finally, if you're not already on an SSD, you might look into spending your money there rather than on memory. But you would first need to benchmark with a high load and find your failure point.