Apache HUNG not reading from socket

[dmwaff]

Anyone else see this issue? Any suggestion on how to troubleshoot this more?

My apache 2.2.9 worker instance on RHEL 5.2 has hung a few time over the last month or so.

I can telnet to the listen port locally and remotely but Apache never serves the request. I have to break out of the telnet.

$ telnet 172.x.x.x 80
Trying 172.x.x.x...
Connected to 172.x.x.x.
Escape character is '^]'.
GET/index.html HTTP/1.1

telnet> quit
Connection closed.

Strace of the running root PID loops the following like it is polling and it is selecting a listen socket but never gets an interrupt telling it to read from the socket.

select(0, NULL, NULL, NULL, {0, 414000}) = 0 (Timeout)
wait4(-1, 0x7fffc35696cc, WNOHANG¦WSTOPPED, NULL) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
wait4(-1, 0x7fffc35696cc, WNOHANG¦WSTOPPED, NULL) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
wait4(-1, 0x7fffc35696cc, WNOHANG¦WSTOPPED, NULL) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
wait4(-1, 0x7fffc35696cc, WNOHANG¦WSTOPPED, NULL) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
wait4(-1, 0x7fffc35696cc, WNOHANG¦WSTOPPED, NULL) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
wait4(-1, 0x7fffc35696cc, WNOHANG¦WSTOPPED, NULL) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
wait4(-1, 0x7fffc35696cc, WNOHANG¦WSTOPPED, NULL) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
wait4(-1, 0x7fffc35696cc, WNOHANG¦WSTOPPED, NULL) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
wait4(-1, 0x7fffc35696cc, WNOHANG¦WSTOPPED, NULL) = 0
select(0, NULL, NULL, NULL, {1, 0} <unfinished ...>

I have to do a full shutdown (apachectl stop) and start to get it to initialize fresh and work again. Graceful; Graceful-stop did not complete; and Restart does not work.

Never see this in lower environments (dev,test,qa) and can't turn on debug in production.

David

[jdMorgan]

Any luck with this problem so far?

(bump)

Jim

[dmwaff]

No luck yet. Happened to two additional HTTP Servers since the post.

Still researching.

[codebrewery]

I'm noticing something very similar intermittently on one of our live servers:

select(0, NULL, NULL, NULL, {0, 94000}) = 0 (Timeout)
wait4(-1, 0x7fff7e92bcbc, WNOHANG|WSTOPPED, NULL) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
wait4(-1, 0x7fff7e92bcbc, WNOHANG|WSTOPPED, NULL) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
wait4(-1, 0x7fff7e92bcbc, WNOHANG|WSTOPPED, NULL) = 0
...

which is making them unresponsive and would be very interested to see if you found out what the problem is!

Ta.

[jdMorgan]

Check into recent platform software upgrades. Servers don't just "decide on their own" to stop working...

Jim