Apache keeps crashing

(This is a long-shot...)

[isorg]

I have a Cobalt Raq4. This is some information about Apache running on it:

Server Version: Apache/1.3.20 Sun Cobalt (Unix) mod_ssl/2.8.4 OpenSSL/0.9.6b PHP/4.1.2 mod_auth_pam_external/0.1 FrontPage/4.0.4.3 mod_perl/1.25
Server Built: Apr 4 2003 16:03:06

For the past month, I have had a problem where for no good or apparent reason, the Apache shuts itself down. It is really annoying. The rest of the server continues fine. I have to access the server by telnet or SSH and from root issue a straightforward /etc/rc.d/init.d/httpd restart command, and it comes back to life.

Normally when you run this command from the command line, it gives a statement saying that it is shutting down Apache, then a few seconds later it gives a statement that it is starting up Apache.

When I try it after the Apache has stopped, there is no confirmation that it has stopped Apache. I am guessing that this is because Apache is already stopped.

I already (before all this started) had created a cron job which restarts the Apache every 2 hours anyway. Sometimes this is successful in restarting it before I notice it has stopped, sometimes not, and I get a warning message from the external server monitoring service. Usually (but not always) after a few hours (maybe 6-8 hours) the Apache starts up again, I am guessing because of the cron job.

Sometimes I have noticed that it can occur if I request a page exactly at the moment when the 2-hourly cron Apache restart is due to take place. Maybe I am over-analysing...

<Sheepish grin> This all started when I was messing with the scripts in /etc/rc.d/init.d ... I am not sure whether I touched the httpd script. You see I had just discovered the SSH script in there and was tinkering with it, and then I thought I'd have a look at the others too...

If any of this makes any sense to anyone, I would be grateful for any ideas :-)

[indiandomain]

i had a similar problem.
i had a script that was pulling data from a 5mb .txt file on the server.
that unstablized my system.

check if you have any script doing many operations.

also whats the message in /var/log/messages?

[bcc1234]

Check httpd-error.log file.
Also enable (or increase verbosity) of loggin for all components that might be failing: php, ssl engine, etc.
Then check all your logs after another crash, that might give you some idea of what's going on.

[isorg]

Captain's Log: this is the last entry before it shuts down. Does that shed any light on it? At 14:30:00 the cron log correctly states that /etc/rc.d/init.d/httpd restart was called as scheduled to restart Apache. This error seems to occur 5 seconds later. Or is the "caught SIGTERM" entry a normal thing, to say that the command to restart was received & implemented?

[Thu Oct 23 14:30:05 2003] [crit] (98)Address already in use: make_sock: could not bind to port 80
[Thu Oct 23 14:30:06 2003] [notice] caught SIGTERM, shutting down

[bcolflesh]

Next time it happens,

ps -ax¦ grep apache

and see which processes are still running - one of them is trying to bind.

[isorg]

What does that mean, when a process is trying to "bind"? What exactly is it trying to do (an in this case not succeeding?) (Sorry for basic question!)

[bcolflesh]

Connect to a port that is already in use by some other process, which in your situation is leading to a fault it can't resolve, so Apache shuts down.

[jdMorgan]

isorg,

From your log as posted above:

Address already in use: make_sock: could not bind to port 80

More than one 'copy' of Apache server is trying to claim port 80 as its port, but that port is already taken.

I'd guess that HTTP port 80 (the 'standard' port) is already taken by the server that you expected to take it, and that some unexpected thing is causing another copy of the server to try to claim the same port.

Make sure your cron-based shutdown/restart sequence is correct to avoid this. (Don't ask *me* how, though. I've never done what you are doing here myself).

Jim

[sugarkane]

This might not have any bearing on your situation, but I had similar problems when handling restarts by cron. Sometimes Apache would not come back up, sometimes it would become unstable, etc etc

I never found the reason for the problem, but I found that using 'httpd reload' rather than 'restart' cured it.

Again, I'm not sure if this is relevant to what you're experiencing, I just thought I'd throw it in ;)

[isorg]

Thanks a million to all of you for helping me out with this one.

I changed "restart" to "reload" and it has not crashed once since. The restart consists of a stop followed immediately by a start. So presumably on a busy server if the start was issued before all the processes had been killed, then this could cause the conflict as described.