That's a strange one. 504 is a "gateway timeout". You should only get this error if your web server is acting as a proxy or gateway to another system. Do you have your web site (or part of your website) set-up as a proxy?
It means that the proxy server timed-out waiting for the system that it is proxying.
Of course, there's nothing preventing some software you have installed on your server from inappropriately returning this status code for some other kind of error.
Errors related to "too many connections" may be as simple a fix as changing your server configuration to allow more connections!
Most web servers do NOT "tune" themselves automatically to the capabilities of your hardware. It is up to YOU, as webmaster, to determine appropriate settings for various limits, and change the configuration files accordingly.
You need to find out exactly what resources (if any) are being stressed, instead of trying to guess at what might improve the situation.