httpd stops by itself

What could be wrong?

[sunroof]

I have a linux server with CentOS and plesk. It's dual xeon 2,8GHz. Httpd stops buy itself every once in a while. Does anyone know why is it happening?

Here is the top output:

-----------
14:30:37 up 1 day, 20:23, 2 users, load average: 1.11, 4.63, 4.95
272 processes: 269 sleeping, 1 running, 2 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 0.1% 0.0% 1.7% 0.0% 0.1% 2.2% 95.6%
cpu00 0.0% 0.0% 2.5% 0.0% 0.5% 3.3% 93.4%
cpu01 0.1% 0.0% 1.7% 0.0% 0.0% 3.3% 94.6%
cpu02 0.3% 0.0% 2.1% 0.0% 0.0% 1.1% 96.2%
cpu03 0.0% 0.0% 0.5% 0.0% 0.0% 0.9% 98.4%
Mem: 2054916k av, 1848512k used, 206404k free, 0k shrd, 61204k buff
1226652k actv, 431408k in_d, 24088k in_c
Swap: 1052248k av, 254292k used, 797956k free 973144k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
25 root 15 0 0 0 0 SW 0.7 0.0 25:43 2 kjournald
1771 root 15 0 540 520 460 S 0.1 0.0 3:49 2 syslogd
1992 qmails 15 0 472 460 376 S 0.1 0.0 3:23 0 qmail-send
11402 sarajevo 16 0 1356 1356 888 S 0.1 0.0 0:28 2 top
1 root 15 0 456 424 396 S 0.0 0.0 0:13 0 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 migration/1
4 root RT 0 0 0 0 SW 0.0 0.0 0:04 2 migration/2
5 root RT 0 0 0 0 SW 0.0 0.0 0:00 3 migration/3
6 root 15 0 0 0 0 SW 0.0 0.0 0:00 1 keventd
7 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
8 root 34 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd/1
9 root 34 19 0 0 0 SWN 0.0 0.0 0:00 2 ksoftirqd/2
10 root 34 19 0 0 0 SWN 0.0 0.0 0:00 3 ksoftirqd/3
13 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
11 root 15 0 0 0 0 SW 0.0 0.0 7:06 0 kswapd
12 root 15 0 0 0 0 SW 0.0 0.0 7:05 0 kscand

----------------------

As you can see there is no httpd process at all... I have to stop httpd and start it again.
When this happens load average is minumum...

Do you know why is this happening?

Thanks

[zCat]

Are httpd processes actually running? You are only showing the upper part of the "top" output, if the processes are inactive they may not be visible.

What do the Apache error logs say?

[sunroof]

Nothing special, just a few "File does not exist" in error_log.

/etc/init.d/httpd status shows that httpds are running...

My server has 2GB of RAM, and sometimes up to 300 parallel users online (forums, blogs etc..).

[lammert]

My server has 2GB of RAM, and sometimes up to 300 parallel users online (forums, blogs etc..).

Hmm, 300 users. Maybe all file descriptors are used up?

Also from your top screenshot:

272 processes: 269 sleeping, 1 running, 2 zombie, 0 stopped

272 processes is for me a sign that many instances of httpd are still running. You should scroll down in the top screen to see the next pages if any httpd instances are left. Apache spreads the load over several forked instances of httpd (maximum 150 in the default configuration if I remember correctly) and each gets just a small piece of work to do. Therefore you won't see them in the default top screen which is sorted on actual processor usage. If an httpd process is waiting for I/O it will jump to the bottom of the list, but the process itself is still active.

[lammert]

On my Linux system, one instance of httpd takes about 30 MB virtual memory. 8 MB is shared. Looking at your current memory configuration of 2GB RAM and a swap space of 1GB, you have a total of 3GB to store the httpd and all other running processes. This limits the number of httpd's to 100 to 150 depending on the size of other processes running, before all available memory is used up. With your 300 on-line users insufficient memory could be the problem that apache stops serving requests. Maybe there is no memory left to fork new httpd servers for new requests and the other enter the idle state because KeepAlive has been turned on.

In this case adding more memory is one solution (if it is technically possible) or at least increase the available swap space. Increasing swap space is not a real solution because swapping on a web-server can cause severe performance problems.

Another solution might be to decrease the KeepAliveTimeout value. This is the time a connection is kept active before it is freed to serve another request. Default is 15 seconds. If this value is too high, many httpd processes will be kept idle causing no processor-load, but clients are also not served.

[sunroof]

First, I wanna thank you.

I followed your suggestions, and upgraded my RAM to 4GB.

Server works fine now, but load average is still sometimes too high.

Do you have any suggestions how to tune in my apache and/or mysql for a better performance, since the most of my pages are dynamic - php/mysql.

I'm gonna try to decrease KeepAliveTimeout too.

Here is my top now: img47.imageshack.us/img47/8388/top8hr.th.gif

I can't believe that although now all 4G of RAM are used.

tnx

[edited by: jdMorgan at 3:35 pm (utc) on Feb. 17, 2006]
[edit reason] De-linked [/edit]

[lammert]

The new top figures look much healthier now. Look at the difference:

2GB: 0.1% user, 1.7% system, 2.2% iowait
4GB: 12.5% user, 4.2% system, 2.3% iowait

The system and I/O wait load are a measure how busy linux is with swapping and keeping the system running. The time spend in user processes is a combination of httpd and other user processes. Also the swap space use is 0k, which is excellent.

If the response at the user side is OK, I would leave it so for a while. A load average around 10 is high, but not unusual or problematic for a multiprocessor machine.

The only thing I see is that after the change in RAM the percentage iowait almost remained the same at 2.3%. Possibly I/O activity is now the bottleneck in your system which could be cause by disk activity. Reducing the log level or disabling some log files might be the next thing to do. Playing with the value of the KeepAliveTimeout decreases the time a httpd connection stays available for another request. Decreasing this value might decrease the amount of simultaneous httpd servers running but with the current figures that is not your problem at the moment.

[sunroof]

Load average is mostly 2-3, and sometimes raises up to 12, but returns in a while to 2-3. But the system works fine, i feel it's so much healthier now too, tnx to new RAM. :)

I'll leave it this way for a while.

I also found some apache tuning tips here [ece.concordia.ca...]

And I set KeepAliveTimeout to 10 (from 15) and MaxClients to 350 (from 120).

Thank you very much.