Server becomes unavailable

[error] [client ::1] Directory index forbidden by Options directive:

[adrianTNT]

Hello.
A dedicated server becomes unavailable and it only gets back online after I request a power cycle, but it goes offline again after around 24 hours or less. It is Fedora OS with Plesk installed.
Anyone knows what can cause this?

My error log is filled with this line multiple times:
[error] [client ::1] Directory index forbidden by Options directive: /var/www/html/

At some point after this error it says that it cannot allocate memory, I assume it uses all server memory?

And the access log is full of this:
"GET / HTTP/1.0" 403 3956 "-" "Apache/2.2.6 (Fedora) (internal dummy connection)"
"GET / HTTP/1.0" 403 3956 "-" "Apache/2.2.6 (Fedora) (internal dummy connection)"

Attached is part of error log, then a bit of access log.
Thanks in advance.

[Wed Oct 13 12:09:58 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 12:10:01 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 12:10:10 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 12:10:11 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 12:10:12 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
FATAL: emalloc(): Unable to allocate 80496387 bytes

___ SERVER WAS OFFLINE HERE (no data for 7 hours) ____


[Wed Oct 13 19:46:30 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Oct 13 19:46:30 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:46:30 2010] [warn] RSA server certificate CommonName (CN) `www.mysite.com' does NOT match server name!?
[Wed Oct 13 19:46:30 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:46:30 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:46:30 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:46:30 2010] [warn] Init: SSL server IP/port conflict: default-208-109-122-203:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:92) vs. webmail:443

(/etc/httpd/conf.d/zz010_psa_httpd.conf:216)
[Wed Oct 13 19:46:30 2010] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!
[Wed Oct 13 19:46:31 2010] [notice] Digest: generating secret for digest authentication ...
[Wed Oct 13 19:46:31 2010] [notice] Digest: done
[Wed Oct 13 19:46:32 2010] [notice] mod_python: Creating 4 session mutexes based on 512 max processes and 0 max threads.
[Wed Oct 13 19:46:32 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:46:32 2010] [warn] RSA server certificate CommonName (CN) `www.mmfiles.com' does NOT match server name!?
[Wed Oct 13 19:46:32 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:46:32 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:46:32 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:46:32 2010] [warn] Init: SSL server IP/port conflict: default-208-109-122-203:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:92) vs. webmail:443
(/etc/httpd/conf.d/zz010_psa_httpd.conf:216)
[Wed Oct 13 19:46:32 2010] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!
[Wed Oct 13 19:46:32 2010] [notice] Apache/2.2.6 (Unix) DAV/2 PHP/5.1.6 mod_python/3.2.8 Python/2.4.4 mod_ssl/2.2.6 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured --

resuming normal operations
[Wed Oct 13 19:47:12 2010] [notice] Graceful restart requested, doing restart
[Wed Oct 13 19:47:12 2010] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Oct 13 19:47:12 2010] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Oct 13 19:47:12 2010] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Oct 13 19:47:12 2010] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Oct 13 19:47:12 2010] [notice] Digest: generating secret for digest authentication ...
[Wed Oct 13 19:47:12 2010] [notice] Digest: done
[Wed Oct 13 19:47:12 2010] [notice] mod_python: Creating 4 session mutexes based on 512 max processes and 0 max threads.
[Wed Oct 13 19:47:12 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:47:12 2010] [warn] RSA server certificate CommonName (CN) `www.mmfiles.com' does NOT match server name!?
[Wed Oct 13 19:47:12 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:47:12 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:47:12 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:47:12 2010] [warn] Init: SSL server IP/port conflict: default-208-109-122-203:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:90) vs. webmail:443

(/etc/httpd/conf.d/zz010_psa_httpd.conf:214)
[Wed Oct 13 19:47:12 2010] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!
[Wed Oct 13 19:47:12 2010] [notice] Apache/2.2.6 (Unix) DAV/2 PHP/5.1.6 mod_python/3.2.8 Python/2.4.4 mod_ssl/2.2.6 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured --

resuming normal operations

__ ERRORS STARTING ___

[Wed Oct 13 19:47:12 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 19:47:12 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 19:47:12 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 19:47:12 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 19:47:12 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 19:47:12 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 19:47:12 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 19:47:12 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 19:47:12 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 19:47:12 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 19:47:12 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 19:47:12 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 19:47:12 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 19:47:12 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 19:47:18 2010] [notice] Graceful restart requested, doing restart
[Wed Oct 13 19:47:18 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Wed Oct 13 19:47:18 2010] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Oct 13 19:47:19 2010] [notice] Digest: generating secret for digest authentication ...
[Wed Oct 13 19:47:19 2010] [notice] Digest: done
[Wed Oct 13 19:47:19 2010] [notice] mod_python: Creating 4 session mutexes based on 512 max processes and 0 max threads.
[Wed Oct 13 19:47:19 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:47:19 2010] [warn] RSA server certificate CommonName (CN) `www.mmfiles.com' does NOT match server name!?
[Wed Oct 13 19:47:19 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:47:19 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:47:19 2010] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Wed Oct 13 19:47:19 2010] [warn] Init: SSL server IP/port conflict: default-208-109-122-203:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:88) vs. webmail:443
(/etc/httpd/conf.d/zz010_psa_httpd.conf:212)
[Wed Oct 13 19:47:19 2010] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!
[Wed Oct 13 19:47:19 2010] [notice] Apache/2.2.6 (Unix) DAV/2 PHP/5.1.6 mod_python/3.2.8 Python/2.4.4 mod_ssl/2.2.6 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured --

resuming normal operations
[Wed Oct 13 19:47:19 2010] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/

AND THE ACCESS LOG:

::1 - - [13/Oct/2010:12:10:10 +0300] "GET / HTTP/1.0" 403 3956 "-" "Apache/2.2.6 (Fedora) (internal dummy connection)"
::1 - - [13/Oct/2010:12:10:11 +0300] "GET / HTTP/1.0" 403 3956 "-" "Apache/2.2.6 (Fedora) (internal dummy connection)"
::1 - - [13/Oct/2010:12:10:12 +0300] "GET / HTTP/1.0" 403 3956 "-" "Apache/2.2.6 (Fedora) (internal dummy connection)"

__ SERVER OFFLINE HERE (no log data for 7 hours) ___

::1 - - [13/Oct/2010:19:47:12 +0300] "GET / HTTP/1.0" 403 3956 "-" "Apache/2.2.6 (Fedora) (internal dummy connection)"
::1 - - [13/Oct/2010:19:47:12 +0300] "GET / HTTP/1.0" 403 3956 "-" "Apache/2.2.6 (Fedora) (internal dummy connection)"
::1 - - [13/Oct/2010:19:47:12 +0300] "GET / HTTP/1.0" 403 3956 "-" "Apache/2.2.6 (Fedora) (internal dummy connection)"
::1 - - [13/Oct/2010:19:47:12 +0300] "GET / HTTP/1.0" 403 3956 "-" "Apache/2.2.6 (Fedora) (internal dummy connection)"

[sublime1]

My first guess is that there is a permissions issue on the log files directory that occurs when logs are rotated, or on the /var/run pidfile that is created when the server is restarted. That fits with the "runs for less than 24 hours".

Then I saw the line

FATAL: emalloc(): Unable to allocate 80496387 bytes

and I tend to think this is the most likely cause -- this means that at some point the server ran out of memory.

But why?

I think the "internal dummy connection" and associated SSL warnings are unrelated and benign, see [wiki.apache.org...]

It's worth tracking down the Directory index forbidden error, although it may be downstream of the others. You could probably get rid of it by adding a simple index.html file in /var/www/html ... unless that's really your main document root.

If you have access to the (Linux) server logs (in /var/log) check the file "messages", looking for either "memory", "alloc" or "oom" -- "oom" means "out of memory" and there's a process called the "oom_killer" that protects core Linux processes from running out of memory by killing non-core ones ... like Apache.

If you can run "free" or "top" (either from Plesk or from a command line) keep an eye on memory usage, swapping, etc. There are a lot of things with caching and the like that tend to nibble away at free memory.

Also, as the server is running, check the free memory -- maybe there's some memory leak or something.

Fedora is kind of ancient, isn't it? There was an RH distro called CentOS, but maybe Fedora is back? As long as it's current and relatively up to date, it's unlikely that this is relevant.

Obviously this is more of a brain dump than an actual diagnosis. But that's what I see. These problems tend to be a major pain to track down. If you don't find it from these, let us know if you do have root access to the command line, in which case it's easier to do some diagnostics.

Tom

[adrianTNT]

Thanks for the reply Tom.
I actually found some other pages that mentioned the same thing, to place an index.html in that directory.
I did and errors are gone, I think problem is gone too.
Memory usage was 40% last night, 30% now; so not going up.

But maybe I can still get some answers ?!

Do you think the "Directory index forbidden" problem was because of some stats/log processing? Because...
- In that /var/www/html directory there was only one folder, the "Awstats"
- I had 3 relative large sites on that server, the erorr only appears in the httpd log when I enable the domain with most traffic.

[sublime1]

Hrrrm...

I am a little skeptical -- there's a clear reason why you were getting the errors -- see the link I put in my prior reply to wiki.apache.org -- it explains most of the errors, but not one.

Why would this error (which is essentially a special case of page not found confounded by SSL issues) cause your server to run out of memory -- I think the running out of memory thing was the likely root cause of the server failure (That "FATAL" message is kind of a tip off :-) )

It's possible that Awstats needs to be turned off (my opinion: use Google Analytics rather than log processing reporting, or in addition to log processing, which can identify site errors better than Google Analytics). If you want to keep it, it sounds like it's in its default configuration, which probably is not right for your site.

But another theory is that you have an older version of Apache that has a memory leak, or perhaps the software of your website has a leak. Are you up to date on all of your patches?

Tom

[adrianTNT]

Sorry, I must have missed that link at first reply. I read it, I don't understand much :) because I am not very familiar with SSL I guess.

Apache and server patches need updates, yes. Hopefull I will change to a new server with CentOS soon. I am avoiding updates because sometimes they caused me more problems than they solved.

The Awstats were enabled by me from Plesk panel, I like that it gives some different details than Google Analytics, for example it shows top clients/ip by KB, I think G Analytics doesn't have that.
e.g: a single IP form RoadRunner USA generated 35GB of traffic this month, when all visitors generated under 400MB each. What the ?!.

Anyway, now only the dummy connection thing remains in access_log, I will leave it for now because server seems stable now.

Thanks Tom.

[jeffatrackaid]

Those plesk mismatch errors can be ignored.

Do you have sysstat installed? If not install it and it will keep track of RAM, CPU and other metrics. You can then cross reference this data with the logs to see if there is something specific triggering the issue.

Sometimes I've seen scripts with memory leaks. The leaks continue to build over time until all swap/RAM is used. At that point the server will lock up.

I would try to identify this issue before migrating, you don't want to just move a problem.

[adrianTNT]

It crashed again so I had another look at that apache wiki page, so finally I edited /etc/httpd/conf/httpd.conf file
and replaced:
CustomLog logs/access_log combined
with this:

SetEnvIf Remote_Addr "::1" loopback
CustomLog logs/access_log combined env=!loopback

Then restarted the server and internal dummy connection was no longer logged in the access_log.
I think it just used too much memory, it added that error to log faster than one second.

Hopefully this will do. Waiting now...