1G virtual private server running LAMP crashes

Munin graph attached

[intellivision]

I'm not a server admin, but grudgingly learned the basics when managed hosting turned out to be a disaster. "If you want it done right, do it yourself." So I built up my own 1GB Debian server at Slicehost, running LAMP stack for a busy Wordpress and phpBB site.

It's gone down about once a week in the 7 months I've had it. I visited this forum and anything on the Web for hints at good my.cnf and apache2.conf configuration practices, and implemented what I found.

_I installed and watch Munin
_I installed and watch mysqltuner.pl

For a time it was stable, but the crashes are back, and more frequent.

Munin's charts from today's two crashes [prssr.com]. I can see what Munin's telling me, but I can't understand why. Is it occasional DOS attacks? This morning's crash was at 3:30 when traffic is light.

Are mysql queries bringing it down? Apache running out of memory? I don't know the answers.

From apache2.conf

Timeout 12
KeepAlive Off
MaxKeepAliveRequests 25
KeepAliveTimeout 2
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 3
MaxSpareServers 6
MaxClients 171
MaxRequestsPerChild 100
</IfModule>
<IfModule mpm_worker_module>
StartServers 2
MaxClients 99
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
HostnameLookups Off

From my.cnf (abbreviated):

key_buffer= 48M
max_allowed_packet= 16M
thread_stack= 64K
thread_cache_size= 64
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover= BACKUP
max_connections = 400
table_cache = 1024
tmp_table_size = 64M
max_heap_table_size= 64M
query_cache_limit = 256k
query_cache_size = 64M

I'm really trying to understand this, but I'm hitting the law of diminishing returns. Insight would be a livesaver. Thank you.

[hugh]

Mysqltuner isn't reliable for recommadations, it's just a guide to testing. If you don't know what your doing use the deafult configuration files for apache and mysql, they should be fine. About the crashes, what do your log files say? How and what is crashing? Is apache segfaulting or the host becoming unresponsive? If apache is crashing what plugins are you running? And what happens if you comment them out? Also what do you mean by a busy site and ddos?

[lammert]

This problem isn't caused by MySQL. If you look at your graphs, just before the server crashes the amount of processes increases suddenly from 85 to 240. MySQL doesn't spawn so much extra processes when load increases, but Apache does. 240-85 = 155, which is near the maximum amount of Apache clients which you have in your Apache config for the prefork configuration (MaxClients=171).

As a rule of thumb, each Apache client takes about 15MB RAM in prefork mode in my experience. This value depends on your scripts, which mods are installed etc so it may be higher or lower in your specific situation, but let's assume this 15MB value. 171 Apache processes of 15MB each need at least 2.5GB RAM. Your VPS only allows 1GB RAM usage, which causes your server to swap heavily and then crash.

You should lower the MaxClients value to something reasonable. With an 1GB server which is also running MySQL a reasonable vaule is probably around 40, which at 15MB per process allocates 600MB for Apache. If more clients connect, they will receive a 50x Server Busy error which is not nice, but still better than a crashing server which doesn't serve any pages until it is rebooted.

The reason for these spikes are probably scrapers or badly behaving bots hitting your site. It could be a targeted DOS attack, but in that case you shouldn't see these spikes, but a continuous overload of your site. Your Apache access_log files should give more information about which client/bot/circumstance caused the heavy load just before the crashes.

[intellivision]

Guys, thanks a ton. lammert, you've led me to look at installed Apache mods, and I think I have too many. I know you can't tell me which ones are extraneous, but if there are any that jump out that just don't need to be in a typical LAMP stack server, please let me know. I'm going to start reading about each and think if my server needs it.

mod_log_config
mod_logio
prefork
http_core
mod_so
mod_alias
mod_auth_basic
mod_authn_file
mod_authz_default
mod_authz_groupfile
mod_authz_host
mod_authz_user
mod_autoindex
mod_cgi
mod_deflate*
mod_dir
mod_env
mod_expires*
mod_headers*
mod_mime
mod_negotiation
mod_php5*
mod_rewrite*
mod_setenvif
mod_status

* I know I need these

Also, I set my MaxClients to 45.

[intellivision]

Another crash with MaxClients at 45. So that'll now be 40.

To get that magic number, I've found a common formula. "An approximation of this number should be derived by dividing the amount of system memory (physical RAM) available by the maximum size of an apache/httpd process..." from [devside.net...] It goes on to say

ps -ylC httpd --sort:rss

Divide number by 1024 to get megabytes

When I run that, I get

S UID PID PPID C PRI NI RSS SZ WCHAN TTY TIME CMD
Z 33 30035 3670 0 80 0 0 0 exit ? 00:00:00 apache2 <defunct>
S 33 30040 3670 0 80 0 4804 81254 - ? 00:00:00 apache2
S 0 3670 1 0 80 0 6168 81132 d_look ? 00:00:24 apache2
S 33 30039 3670 0 80 0 10196 81544 - ? 00:00:00 apache2
S 33 30041 3670 0 80 0 12348 81781 - ? 00:00:00 apache2
S 33 30033 3670 0 80 0 13632 81745 - ? 00:00:00 apache2
S 33 30038 3670 0 80 0 14148 81717 192483 ? 00:00:00 apache2
S 33 30036 3670 0 80 0 14840 81845 - ? 00:00:00 apache2
S 33 30013 3670 0 80 0 15524 81836 - ? 00:00:00 apache2
S 33 30019 3670 0 80 0 30376 85794 - ? 00:00:00 apache2

The "RSS" column suggests that my Apache instances are consuming around 14MB of ram each, except for one. lammert you were dead on.

Changed to:

MaxKeepAliveRequests 25
KeepAliveTimeout 1

And added:

<Directory />
Options FollowSymLinks
</Directory>

<Directory />
AllowOverride None
</Directory>

ExtendedStatus Off

Now I need to look into reducing the modules apache loads so I can get that 14MB down, and raise MaxClients.