Too many idle Apache threads. Server becomes no response.

[xinper]

Hi,
This morning our web site could not be accessible. When I login via SSH, I found that 150 Apache threads were running in the server. After I restarted httpd service, the problem was solved. The same problem happened several times already. Most of those Apache threads were idle, but they weren¡¯t closed. Any Suggestion? Thanks.

OS: CentOS 4
Apache 1.3
PHP 4

Our Apache configure:
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
MinSpareServers 5
MaxSpareServers 10
StartServers 5
MaxClients 150

The command ¡°ps aux ¦ grep httpd¡± output in this morning:

nobody 5996 0.2 4.4 52652 45636? S Jul11 4:14 /usr/local/apache/bin/httpd
nobody 5997 0.3 4.3 52368 45316? S Jul11 4:36 /usr/local/apache/bin/httpd
nobody 5998 0.2 4.4 53352 46336? S Jul11 3:48 /usr/local/apache/bin/httpd
nobody 5999 0.2 2.8 36276 29220? S Jul11 3:52 /usr/local/apache/bin/httpd
nobody 6000 0.2 4.2 51240 44188? S Jul11 4:02 /usr/local/apache/bin/httpd
nobody 6001 0.2 1.2 20308 13292? S Jul11 3:35 /usr/local/apache/bin/httpd
nobody 6003 0.3 2.0 30140 21024? S Jul11 4:41 /usr/local/apache/bin/httpd
nobody 6004 0.3 2.9 37956 30908? S Jul11 4:39 /usr/local/apache/bin/httpd
nobody 6005 0.2 2.4 32612 25420? S Jul11 3:38 /usr/local/apache/bin/httpd
nobody 6504 0.2 4.4 52836 45780? S Jul11 4:06 /usr/local/apache/bin/httpd
nobody 21668 0.0 0.8 16748 9180? S 02:22 0:01 /usr/local/apache/bin/httpd
nobody 21882 0.0 0.5 13008 5408? S 02:29 0:00 /usr/local/apache/bin/httpd
nobody 21884 0.0 0.5 13008 5416? S 02:29 0:00 /usr/local/apache/bin/httpd
nobody 21885 0.0 0.5 13008 5424? S 02:29 0:00 /usr/local/apache/bin/httpd
nobody 21886 0.0 0.5 13008 5416? S 02:29 0:00 /usr/local/apache/bin/httpd
¡­
¡­
nobody 22184 0.0 0.5 13124 5544? S 02:30 0:00 /usr/local/apache/bin/httpd
nobody 22186 0.0 0.5 13176 5524? S 02:30 0:00 /usr/local/apache/bin/httpd
nobody 22187 0.0 0.5 13008 5416? S 02:30 0:00 /usr/local/apache/bin/httpd
nobody 22188 0.0 0.5 13008 5424? S 02:30 0:00 /usr/local/apache/bin/httpd
nobody 22189 0.0 0.8 16692 9172? S 02:30 0:00 /usr/local/apache/bin/httpd

[vincevincevince]

Try turning off keep_alive in your httpd.conf file, and failing that - check up on your database settings.

To me it sounds like either:
1) database driven process hanging and keeping the apache thread open during long or locked database operation - try repairing databases as well here
2) cron-driven process which invokes your httpd isn't finishing for some reason - perhaps badly written command for some reason waiting for input?

Check where your open sockets are connected - they may point to a site or similar which is doing something.

Though - could they be DOS / hacking threads in progress?