All the 'R's in server-status

website stopping and server-status showing all processes as R

[kevintynfron]

Hi, not sure if this problem goes here or under linux.

I've inherited a web server to look after which is fine for most of the time, then just seems to stop for a couple of minutes.

Looking at the /server-status page, I saw that all 50 processes were 'R' except one 'W' which I assume is the one writing the status to me.

Current Time: Wednesday, 19-Nov-2008 11:55:00 GMT
Restart Time: Sunday, 16-Nov-2008 03:40:58 GMT
Parent Server Generation: 1
Server uptime: 3 days 8 hours 14 minutes 1 second
50 requests currently being processed, 0 idle workers
RRRRRRRRRRRWRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR

Does the R signify reading from the client, or from the site or both?

I'm pretty new to linux / apache, so the mods I've done this far are:

• changed Max_processes from 10 to 20, then 50
  
• switched on KeepAliveTimeout at 2 seconds
  
• enabled /server-status with password protection
  
• enabled expired and last-modified headers for some directories
  
• enabled defalting of css and js files
  

Another post that was similar suggested a DOS attack. Is this likely? Any other ideas/comments?

Cheers,
Kevin Jones

top - 14:35:00 up 27 days, 3:16, 1 user, load average: 0.04, 0.10, 0.13
Tasks: 53 total, 1 running, 52 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1% us, 0.1% sy, 0.0% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 8207044k total, 390904k used, 7816140k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28624 root 15 0 6588 1520 1080 S 0.0 0.0 0:00.08 bash
27771 root 15 0 15144 1312 1008 S 0.0 0.0 0:00.00 courierlogger
27781 root 23 0 15012 1148 856 S 0.0 0.0 0:00.00 courierlogger
27789 root 16 0 15140 1308 1008 S 0.0 0.0 0:00.00 courierlogger
27806 root 18 0 15008 1144 856 S 0.0 0.0 0:00.00 courierlogger
27769 root 15 0 17296 972 656 S 0.0 0.0 0:00.00 couriertcpd
27779 root 18 0 17296 964 648 S 0.0 0.0 0:00.00 couriertcpd
27787 root 15 0 17296 972 656 S 0.0 0.0 0:00.01 couriertcpd
27804 root 21 0 17296 964 648 S 0.0 0.0 0:00.00 couriertcpd
19877 root 16 0 9740 968 532 S 0.0 0.0 0:00.73 crond
32166 root 15 0 163m 18m 9964 S 0.0 0.2 1:34.04 httpd
11340 apache 15 0 167m 16m 5272 S 0.0 0.2 0:00.18 httpd
11471 apache 16 0 163m 11m 2464 S 0.0 0.1 0:00.03 httpd
11507 apache 16 0 163m 11m 2496 S 0.0 0.1 0:00.04 httpd
11514 apache 16 0 163m 11m 2556 S 0.0 0.1 0:00.02 httpd
11516 apache 16 0 163m 11m 2332 S 0.0 0.1 0:00.01 httpd
11549 apache 15 0 163m 11m 2440 S 0.0 0.1 0:00.01 httpd
11573 apache 16 0 163m 11m 2280 S 0.0 0.1 0:00.01 httpd
11613 apache 15 0 163m 11m 2272 S 0.3 0.1 0:00.02 httpd
11614 apache 15 0 163m 11m 2276 S 0.0 0.1 0:00.01 httpd
11691 apache 15 0 163m 11m 2188 S 0.0 0.1 0:00.00 httpd
28218 root 16 0 97300 9072 4412 S 0.0 0.1 0:01.04 httpsd
28222 psaadm 16 0 149m 32m 25m S 0.0 0.4 0:47.95 httpsd
 7917 psaadm 16 0 150m 25m 17m S 0.0 0.3 0:44.45 httpsd
 1 root 15 0 4808 660 548 S 0.0 0.0 0:05.85 init
 7769 root 16 0 30296 2100 1612 S 0.0 0.0 0:15.67 monit
 5371 mysql 16 0 141m 39m 5452 S 0.0 0.5 142:45.71 mysqld
 5328 root 18 0 6432 1192 972 S 0.0 0.0 0:00.00 mysqld_safe
27999 named 16 0 83424 3676 1764 S 0.0 0.0 0:00.08 named
19886 mailman 16 0 50328 7776 936 S 0.0 0.1 0:00.01 python
19911 mailman 15 0 51456 9.8m 3180 S 0.0 0.1 0:08.99 python
19912 mailman 15 0 51468 9.8m 3180 S 0.0 0.1 0:09.29 python
19913 mailman 16 0 51420 9.8m 3180 S 0.0 0.1 0:09.01 python
19914 mailman 16 0 51412 9.8m 3180 S 0.0 0.1 0:09.19 python
19915 mailman 16 0 51456 9.8m 3180 S 0.0 0.1 0:09.19 python
19916 mailman 16 0 51460 9.9m 3180 S 0.0 0.1 0:09.51 python
19917 mailman 16 0 51456 9.8m 3180 S 0.0 0.1 0:09.18 python
19918 mailman 15 0 51456 9.8m 3180 S 0.0 0.1 0:00.13 python
19539 qmailq 15 0 2556 380 300 S 0.0 0.0 0:01.96 qmail-clean
19537 root 15 0 2600 412 300 S 0.0 0.0 0:00.10 qmail-lspawn
10172 qmailr 18 0 9524 1056 812 S 0.0 0.0 0:00.00 qmail-remote
10173 qmailr 16 0 15188 1340 1032 S 0.0 0.0 0:00.00 qmail-remote.mo
19538 qmailr 15 0 4316 2152 296 S 0.0 0.0 0:03.63 qmail-rspawn
19534 qmails 15 0 2840 712 392 S 0.0 0.0 0:22.67 qmail-send
 7601 root 16 0 18612 1552 1092 S 0.0 0.0 0:00.00 sftp-server
19536 qmaill 15 0 2560 496 416 S 0.0 0.0 0:06.15 splogger
19558 root 16 0 23000 1228 804 S 0.0 0.0 0:43.96 sshd
28532 root 16 0 42500 3196 2256 S 0.0 0.0 0:01.48 sshd
 7599 root 16 0 42656 3256 2240 S 0.0 0.0 0:00.01 sshd
19664 root 16 0 3660 580 460 S 0.0 0.0 0:12.86 syslogd
11341 root 15 0 7228 1164 876 R 0.0 0.0 0:00.14 top
 7761 root 16 0 106m 10m 5580 S 0.0 0.1 0:01.76 wdcollect
27857 root 16 0 8748 864 688 S 0.0 0.0 0:00.04 xinetd

[ajcoon]

Take a look at your netstat:
netstat -anp¦grep apache¦grep -v LISTEN¦wc -l

Does this # match the # of R's in your server-status page?

[kevintynfron]

This seems to an issue with the clients. Either they're behaving badly, or there's a DOS attack. The number of R processes comes and goes during normal use, and I can see that when the server was set up to only have 10 processes max, it could easily saturate them with 10 'R' processes. The example I posted at the start of this thread had maxprocesses set to 50, and I've upped it to 75.

An 'R' process seems to be waiting in a FIN_WAIT_2 status, and there's notes on that here [httpd.apache.org].

The answer would appear to be to use the Timeout directive. The Apache Docs [httpd.apache.org] detail how it cuts off connections that are taking too long.

At the moment timeout is set to 120 seconds. I've switched server-status to use extended mode, so I can see a bit more detail. Watching the processes, every now and a gain, I'll get one that is stuck on 'R'. After 120 seconds, it gets killed off, which is what I want.

Monitoring the processes (although only for about 10 minutes), 'normal' requests come in generally below 10 seconds (well, I don't see them as R processes), but there's a few that can take up to 30 seconds, so I think I'll try lowering it to 30 seconds for now.

Presumably the downside would be if there's a bandwidth issue, there could be a problem. Any ideas how files are uploaded to an apache server? Timeout will also affect "The amount of time between receipt of TCP packets on a POST or PUT request" So I may get problems if somebody's uploading stuff on a slow link.

Unfortunately, one of the tricky things about this is that I can't actually tell if I've solved it, just that it hasn't happened for some times, and as the problem was random to start with, I'll never really know.

Cheers,
Kevin

[jdMorgan]

If you expect that some connections will remain active for 30 seconds, then don't set the timeout below twice that, just for safety.

If your clients upload using HTTP, then you'll need to allow for the biggest allowable upload on the slowest-possible connection, i.e. a 56kbps modem running at half speed (and possibly below that), and whatever your maximum allowable upload filesize is.

You migth want to look into mod_dos_evasive as well. It might allow you to set longer timeouts, run fewer processes, and yet survive request floods if someone is really hitting you with DOS attempts.

Jim