Tuning Apache

Modules Prefork, Worker and KeepAlive

[Mark Wallace]

Hi, on my httpd.conf file I changed the maxclients and serverlimit values from 50 to 150 because the maxclients limit had been reached and I was wondering if the other values could also be fine tuned.

The server has 1GB of RAM and normally uses from 600 to 800 MB.

These are my current setings on the httpd.conf:

KeepAlive Off

MaxKeepAliveRequests 100

<IfModule prefork.c>
StartServers 1
MinSpareServers 1
MaxSpareServers 5
ServerLimit 150
MaxClients 150
MaxRequestsPerChild 4000
</IfModule>

<IfModule worker.c>
StartServers 1
MaxClients 150
MinSpareThreads 1
MaxSpareThreads 4
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>

[sublime1]

Mark --

The first thing you need to do is figure out which of the MPM modules your server is using, if on apache 2.2, either prefork, event, or worker.

Use this command to find out which module is compiled in. Other MPM configurations are not relevant.

apache2ctl -l

Now you have to figure out how much memory a typical client/child instance uses. Various sites I manage use either Munin or MRTG to provide nice charts of usage over time, but to get at the per-instance memory usage, (assuming a Linux server) tools like `top` or `free` are useful. For me, sorting top by memory usage tends to group the things I care about together (Shift-M in my top, others may work differently).

Here's a sample from one of our servers, running a Drupal (PHP) application:

 PID USER   PR NI VIRT RES SHR S %CPU %MEM  TIME+ COMMAND                                              
23332 www-data 20  0 88892 56m 3888 S  0 3.8  0:03.64 apache2                                              
23156 www-data 20  0 84732 52m 4156 S  0 3.5  0:02.40 apache2                                              
23331 www-data 20  0 83076 50m 3884 S  0 3.4  0:01.02 apache2                                              
 2266 mysql   20  0 135m 47m 3756 S  2 3.2 13085:25 mysqld                                               
23123 www-data 20  0 70320 40m 5504 S  0 2.7  0:10.37 apache2                                              
23342 www-data 20  0 68720 36m 3964 S  27 2.5  0:00.80 apache2                                              
 2081 root   20  0 40904 12m 6672 S  0 0.8  0:25.92 apache2                                              
23325 www-data 20  0 41784 8048 1572 S  0 0.5  0:00.03 apache2                                              
23335 www-data 20  0 41036 7324 1476 S  0 0.5  0:00.00 apache2                                              
23336 www-data 20  0 41036 7304 1464 S  0 0.5  0:00.00 apache2                                              
23338 www-data 20  0 41036 7120 1336 S  0 0.5  0:00.00 apache2                                              
23339 www-data 20  0 41036 7120 1336 S  0 0.5  0:00.00 apache2                                              

The key number is RES (resident in memory) which you can see ranges from a high of 56m to a low of 7120. If a bot comes along and decides to pummel a server, I need to make sure that from all the spare memory on the server, how many simultaneous instances could my server support before running out of memory and moving into swapping. The problem is, the amount of memory needed varies and depends on the application settings, and all sorts of other stuff.

One key thing is that as an optimization, Apache keeps an instance around to respond to some number of requests before killing it and starting another. This is set by the MaxRequestsPerChild value. Keeping children around is good since it can cache some stuff and is already warmed up and waiting, but bad because the longer it's around, the bigger it will get. I have set the MaxRequestsPerChild at 400, which, based on my observation makes the average child use about 40MB of RAM (omigod, PHP and Drupal are PIGS!).

If I have 1GB of free RAM (after MySQL and the operating system and whatever else is running) then I should be able to set MaxClients to about 25. I would probably set it to 20 to be conservative, since there are other things the OS caches and more memory is used by other parts of the system when it's under heavy load.

I would strongly advise you you look for further advice on how to set apache. Running out of memory means your server is effectively dead. Much better to be defensive.

Tom

[Mark Wallace]

Hi Tom,

First of all I'd like to thank you for your detailed and helpful response.

These are the compiled in modules:
core.c
prefork.c
http_core.c
mod_so.c

This is the top result using Shift + M (great tip btw)

Link to screencap: [i52.tinypic.com ]

Tasks: 56 total, 1 running, 54 sleeping, 0 stopped, 1 zombie
Cpu(s): 0.1%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1048576k total, 593436k used, 455140k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached


PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1645 root 16 0 273m 77m 15m S 0 7.6 0:51.90 java
32490 named 18 0 131m 27m 2140 S 0 2.7 0:03.46 named
9760 apache 15 0 54176 27m 4056 S 0 2.7 0:00.48 httpd
7231 apache 15 0 53464 26m 4088 S 11 2.6 0:03.82 httpd
9660 apache 18 0 53456 26m 4052 S 0 2.6 0:00.40 httpd
7665 apache 15 0 53452 26m 4036 S 0 2.6 0:00.45 httpd
1668 root 15 0 44636 23m 9668 S 0 2.3 2:36.75 httpd
32686 mysql 15 0 138m 22m 4816 S 1 2.2 9:04.86 mysqld
9767 apache 18 0 45352 16m 2048 S 0 1.6 0:00.03 httpd
9774 apache 15 0 45104 16m 2052 S 0 1.6 0:00.05 httpd
9775 apache 18 0 45216 16m 2044 S 0 1.6 0:00.00 httpd
10042 apache 15 0 44772 15m 1848 S 0 1.5 0:00.00 httpd
1481 cyrus 18 0 10888 2732 1612 S 0 0.3 0:01.21 cyrus-master
1541 cyrus 21 0 32240 2400 1996 S 0 0.2 0:00.00 lmtpd
24048 root 15 0 7972 2204 1804 S 0 0.2 0:00.02 sshd
1536 cyrus 18 0 32444 2172 1760 S 0 0.2 0:00.00 imapd
1543 cyrus 23 0 32444 2172 1760 S 0 0.2 0:00.00 imapd
1535 cyrus 18 0 32444 2160 1752 S 0 0.2 0:00.00 imapd
1542 cyrus 22 0 32444 2160 1752 S 0 0.2 0:00.00 imapd
1562 cyrus 24 0 32444 2160 1748 S 0 0.2 0:00.00 imapd
1564 cyrus 24 0 32444 2160 1748 S 0 0.2 0:00.00 imapd
1709 cyrus 20 0 32444 2160 1748 S 0 0.2 0:00.00 imapd
1711 cyrus 20 0 32444 2160 1748 S 0 0.2 0:00.00 imapd
1715 cyrus 23 0 32444 2160 1748 S 0 0.2 0:00.00 imapd
1717 cyrus 18 0 32444 2160 1748 S 0 0.2 0:00.00 imapd
1710 cyrus 20 0 32444 2156 1748 S 0 0.2 0:00.00 imapd
1714 cyrus 19 0 32444 2156 1748 S 0 0.2 0:00.00 imapd
1539 cyrus 19 0 32144 2136 1736 S 0 0.2 0:00.00 pop3d
1547 cyrus 25 0 32144 2136 1736 S 0 0.2 0:00.00 pop3d
1712 cyrus 19 0 32144 2136 1736 S 0 0.2 0:00.00 pop3d
9556 cyrus 18 0 32144 2136 1736 S 0 0.2 0:00.00 pop3d
9579 cyrus 18 0 32144 2136 1736 S 0 0.2 0:00.00 pop3d
9692 cyrus 18 0 32144 2136 1736 S 0 0.2 0:00.00 pop3d
1540 cyrus 20 0 32144 2132 1736 S 0 0.2 0:00.00 pop3d
1563 cyrus 24 0 32144 2132 1736 S 0 0.2 0:00.00 pop3d
1630 postfix 18 0 6828 1780 1432 S 0 0.2 0:00.01 qmgr
7571 postfix 15 0 6764 1752 1420 S 0 0.2 0:00.01 tlsmgr

Edits: Added top screencap, found out the compiled modules.

[sublime1]

OK, so your server is using the "prefork" MPM and this is the section you should be editing. It looks like each child apache process uses about 15-30MB of RAM when in use, and with 8 child instances (and the root/parent) apache is using about 250MB. You have 450MB of free RAM So you could probably set MaxChildren to 16 - 20. From the screen cap your load average is nice and low. So chances are you don't need to go even that high - having some spare headroom is a lot better than running out of memory!

Wait for a time when your site is getting a lot of traffic (bots, typically) and use the

sudo apache2ctl status

command to see how many processes are in use and what they are doing (you'll need to enable mod_status if it isn't already, but it's worth it -- see the doc for Apache).

Having a higher MaxClients/MaxChildren setting has no value if your server never uses them, or would crawl to a halt in cases where they are being called for.

[Mark Wallace]

Thank you for you help.

I started this thread because during the busyest hours when I use FTP or SSH I notice allot of lag and it seems that the server is too busy and breaking connection every second, although the websites don't get much slower.
I thought that somehow this would be a RAM related problem but it seems that it's not the case.

Any idea on what could be causing this lag?

[sublime1]

In general, a child will be created as needed, new spares get "warmed up" based on the "spare" settings. If you're seeing a lot of thrashing, children coming and going, here are a couple of ideas.

They will accept up to MaxRequestsPerChild requests before they kill themselves or if the number of active children falls so there are more than MaxSpareServers running. If you have the memory, MaxSpareServers should probably be set close to MaxClients -- this way, children will continue to be available for requests and not get killed needlessly. The only reason (as I understand) that a child doesn't just live forever is to deal with various forms of memory leaks -- a kind of self-defense mechanism.

I am pretty sure that you want KeepAlive on -- this allows a client (browser) to re-use an established connection for multiple requests (e.g. for images, css, js that are needed to serve a given page. In thinking about it, this alone may be the cause of some performance issues or processes coming or going more often than expected.

I am working on figuring this stuff out for a situation that sounds similar to yours, and the part I am not clear on is why (in my case) I need to set MaxClients so low compared to past server configurations -- if I don't the server runs out of memory. So here's where I would suggest you set things, but it's a guess that you should monitor using the status monitor (apache2ctl status).

So maybe something like:

# Change to on! 
KeepAlive On
# Isn't this moot is KeepAlive is off?
MaxKeepAliveRequests 100 

<IfModule prefork.c> 
# Warm up 5 servers at start
StartServers 5
# warm up 2 more once needed
MinSpareServers 2
# Kill off a few if we have more than 10 idle servers
MaxSpareServers 10
# Set as high as you need to handle high-load cases while still leaving system memory free
ServerLimit 15
MaxClients 15
MaxRequestsPerChild 4000 
</IfModule> 

Let me know how this works out.

Tom

[Mark Wallace]

Ok Tom I will, I'll do that when I have some time free to closely monitor and analyze the sever performance and be on top of things just in case.

About the MaxClients settings, about a month ago it was set to 50 and I got an error message saying that the MaxClients limit had been reached, it's ok to reduce it to 15?

Best regards,
Mark

[sublime1]

Mark --

Reaching MaxClients means you have a lot of load on your server. It happens often from rogue bots that fire off a stream of requests faster than your server can respond. Additional requests will be queued according to the setting of ListenBacklog, defaulting to 500 (511, actually, for some reason). This means that once you have MaxClients working away, up to 500 requests waiting and getting served one-by-one after that.

So, you should set MaxClients to the highest value possible that your server doesn't run out of RAM and also to a point where server performance doesn't start getting worse.

The key point in all of this, at least for my case, was that the default settings for Apache would allow my server to keep allocating more child servers until the server ran out of memory (and/or started swapping), and that is a Very Bad Thing.

In my case, the "OOM Killer" kicked in -- this is part of the Linux kernel that protects the operating system from going down by killing processes to free memory. In my case, it always picked the MySQL server, which instantly took down the site, but once I tuned it so it wouldn't pick MySQL and made it prefer to kill Apache child processes, but the user experience was terrible since Unix and Apache we're fighting each other -- Unix killing things, Apache starting them up.

So setting MaxClients correctly, is a simple matter of doing some math: how much does the average client use, and how much do you have. Here's a good article on this and other related topics.

[howtoforge.com...]

I think one thing to consider is that web applications do a lot more than they used to, and languages like PHP, Java, Ruby are bigger and slower than C or C++. Settings like 256 for MaxClients made sense a few years ago, but today the number needs to be smaller, especially when you're running your database, mail server and other stuff all on one box.

And now, since all I really care about is serving actual users and well-behaved bots, I am thinking: why not set that "ListenBacklog" number lower, say to 50. When that limit is reached, the server will respond with a 503 Busy response, and my guess is that is typically enough to shoo off the stupidest and least well-behaved bots quickly while still handling bursts gracefully.

Tom

[Mark Wallace]

Great article, I'll analyse it asap and post back any results.

I set Keepalive to On and the memory use went from ~ 550MB to almost 800MB.

Edit: KeepAliveTimeout was set to 15, I reduced it to 2 and the memory usage seems to have normalized

[sublime1]

A fine article from a few years ago on this topic: [webmasterworld.com...] (complements of jdMorgan, our fearless forum moderator!), but 15 seconds should be fine for KeepAlive. The reason memory use would jump is that connections are being kept open (blocking others) for a few seconds meaning more child servers are needed. 2 seconds seems short to me, but maybe 5 would be fine.

The use-case of KeepAlive is that a page loads in a browser, and the moment afterward the same browser, if using HTTP 1.1, will use the same connection to request other assets like CSS, JS, and images, all on the same connection. I assume the timeout starts counting from the completion of the last request. So maybe shorter is fine.

Interesting!

Tom

[phranque]

the apache tuning documentation:

Apache Performance Tuning - Apache HTTP Server:
http://httpd.apache.org/docs/2.2/misc/perf-tuning.html [httpd.apache.org]

another reference i have found useful in the past, covering some of the same topics discussed here and a few others, including some performance testing tools.

Apache Performance Tuning:
http://www.devside.net/articles/apache-performance-tuning [devside.net]

[sublime1]

Thanks!

A great quote from the second link: "Premature optimization is the root of all evil" :-)

Thanks -- the 2 second keepalive seems to be right for the prefork module -- a good explanation there.

[Mark Wallace]

The server has been running for the last 24 hours with the following base configuration:

StartServers 3
MinSpareServers 3
MaxSpareServers 10
ServerLimit 50
MaxClients 50
MaxRequestsPerChild 4000

I've set KeepAlive to one second beacuse it does occupy a considerable amount of RAM with prefork, I noticed a small difference even changing it from 2 seconds to 1.

But enabling KeepAlive had a very postive effect on my website's performance, they load faster and much smoother, raising the StartServers and MinSpareServers to from 1 to 3 also contributed to this.

Performance increased and so did RAM usage, from a steady 550 - 650 MB to 600 - 850 MB, but I rather keep performance for now.

An interesting note regarding MaxRequestsPerChild:
For KeepAlive requests, only the first request is counted towards this limit. In effect, it changes the behavior to limit the number of connections per child.

[httpd.apache.org...]

[sublime1]

Cool stats -- yeah, as a result of you question I have been fiddling around on a couple of the sites I manage and have found similar results. Thanks for asking :-)

I suspect that the MaxRequestsPerChild thing is a vestige of the past where Request and Connection were pretty much the same thing. The key thing with MaxRequestsPerChild is that it is a mostly a self-protective mechanism by Apache, recognizing that between caching, memory leaks and other stuff, it's probably a good idea to kill off a connection every so often.

[Mark Wallace]

I tuned down the settings a bit and now RAM use rarely spikes past 800MB. There's no noticeable loss in performance.

StartServers 2
MinSpareServers 2
MaxSpareServers 6
ServerLimit 100
MaxClients 100
MaxRequestsPerChild 4000

Many thanks to Tom and phranque for all the help provided.

[sublime1]

Mark --

Back to first principles: your configuration could take down your site!

Do you really have 1.6GB of free memory? Not according to your screen capture which said you had about 400MB free.

Looking at one of the apache instanced from your top display:

9767 apache 18 0 45352 16m 2048 S 0 1.6 0:00.03 httpd 

One client is using 16mb of RAM. Some more, some less. If you site gets loaded down, Apache will keep allocating more clients, up to 100 until the operating system is out of RAM, at which point it will start pushing stuff onto swap or killing processes. There's another word for this: server down.

Of course your site will seem to be fine under normal load, but if the 400MB free number is right, and the average apache client uses 16MB, your MaxClients should be no more than 25 (400/16).

(You're welcome to calculate the actual average size of clients under load, and the actual amount of free memory -- maybe you'll arrive at a different number. But I guarantee, with what you have posted here, 100 is too high for MaxClients)

This is not a drill. Trust me: I learned this the hard way :-)

Tom

[Mark Wallace]

You're right! And finally I understood how to make that calculation. :)

I've changed MaxClients to 25.