Very frustrating: Apache process locks up?

[cranky7]

Hi all,

We're at our wits end here. For ages (years) we have been struggling with a random & reoccurring issue: once in a while users seemingly get locked out of our site.

When this happens, all attempts by the user to access our site (from any page) will result in an endless hang. The page looks like it is loading, but nothing happens. The only 3 ways to solve this are:
1) Restart apache or kill their process
2) Use a new browser
3) Wait for a few hours and come back.

When this happens, the site continues to run fine for all our other visitors. The issue seems to be unrelated to server-load or any other variables. It may happen to me personally once every couple weeks, with others having similar problem rates.

Our site is fairly popular, ranked in the top couple thousand by Alexa. We are running a single Redhat web server, with 8GB of ram and 2x 3.0 Xeon processors. However, the issue has happened using other machines in the past.

Given how often we run into this problem, we are amazed by the fact that we have been unable to find any other similar problems online. Perhaps we are just searching the wrong thing, but we desperately hope someone here may be able to help us out.

Any help would be greatly appreciated!

Below is most of our httpd.conf file:

ServerTokens OS
ServerRoot "/etc/httpd"
PidFile run/httpd.pid
Timeout 20
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 5

<IfModule prefork.c>
 StartServers 10
 MinSpareServers 5
 MaxSpareServers 20
 ServerLimit 300
 MaxClients 300
 MaxRequestsPerChild 200
</IfModule>


<IfModule worker.c>
 StartServers  2
 MaxClients  150
 MinSpareThreads 25
 MaxSpareThreads 75
 ThreadsPerChild 25
 MaxRequestsPerChild 0
</IfModule>

Listen 80

LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_alias_module modules/mod_authn_alias.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule logio_module modules/mod_logio.so
LoadModule env_module modules/mod_env.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule expires_module modules/mod_expires.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule info_module modules/mod_info.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule cache_module modules/mod_cache.so
LoadModule suexec_module modules/mod_suexec.so
LoadModule mem_cache_module modules/mod_mem_cache.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule version_module modules/mod_version.so

Include conf.d/*.conf
ExtendedStatus On
TraceEnable off
User apache
Group apache

ServerAdmin root@localhost
ServerName server#*$!X:80

UseCanonicalName Off
DocumentRoot "/home/virtual2/domain.com/webroot/htdocs"

<Directory />
 Options FollowSymLinks
 AllowOverride None
</Directory>

<Directory "/home/virtual2/domain.com/webroot/htdocs">
 Options FollowSymLinks
 AllowOverride None
 Order allow,deny
 Allow from all
</Directory>

<IfModule mod_userdir.c>
 UserDir disable
</IfModule>

DirectoryIndex index.html index.php index.htm default.htm index.html.var

AccessFileName .htaccess

<Files ~ "^\.ht">
 Order allow,deny
 Deny from all
</Files>

<Directory ~ "\.svn">
 Order allow,deny
 Deny from all
</Directory>

TypesConfig /etc/mime.types
DefaultType text/plain

<IfModule mod_mime_magic.c>
 MIMEMagicFile conf/magic
</IfModule>

HostnameLookups Off

EnableSendfile off

ErrorLog "|/usr/sbin/cronolog --symlink=/var/log/http_error_log /var/log/apache/%Y.%m/http_error.%Y-%m-%d"

LogLevel warn
LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
CustomLog "|/usr/sbin/cronolog --symlink=/var/log/http_access_log /var/log/apache/%Y.%m/http_access.%Y-%m-%d" combined

ServerSignature On

Alias /icons/ "/var/www/icons/"

<Directory "/var/www/icons">
 Options Indexes MultiViews
 AllowOverride None
 Order allow,deny
 Allow from all
</Directory>

<IfModule mod_dav_fs.c>
 DAVLockDB /var/lib/dav/lockdb
</IfModule>

ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

<Directory "/var/www/cgi-bin">
AllowOverride None
 Options None
 Order allow,deny
 Allow from all
</Directory>

IndexOptions FancyIndexing VersionSort NameWidth=* HTMLTable

AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip
AddIconByType (TXT,/icons/text.gif) text/*
AddIconByType (IMG,/icons/image2.gif) image/*
AddIconByType (SND,/icons/sound2.gif) audio/*
AddIconByType (VID,/icons/movie.gif) video/*
AddIcon /icons/binary.gif .bin .exe
AddIcon /icons/binhex.gif .hqx
AddIcon /icons/tar.gif .tar
AddIcon /icons/world2.gif .wrl .wrl.gz .vrml .vrm .iv
AddIcon /icons/compressed.gif .Z .z .tgz .gz .zip
AddIcon /icons/a.gif .ps .ai .eps
AddIcon /icons/layout.gif .html .shtml .htm .pdf
AddIcon /icons/text.gif .txt
AddIcon /icons/c.gif .c
AddIcon /icons/p.gif .pl .py
AddIcon /icons/f.gif .for
AddIcon /icons/dvi.gif .dvi
AddIcon /icons/uuencoded.gif .uu
AddIcon /icons/script.gif .conf .sh .shar .csh .ksh .tcl
AddIcon /icons/tex.gif .tex
AddIcon /icons/bomb.gif core
AddIcon /icons/back.gif ..
AddIcon /icons/hand.right.gif README
AddIcon /icons/folder.gif ^^DIRECTORY^^
AddIcon /icons/blank.gif ^^BLANKICON^^

DefaultIcon /icons/unknown.gif

ReadmeName README.html
HeaderName HEADER.html

IndexIgnore .?* *~ *# HEADER* README* RCS CVS *,v *,t
AddLanguage ca .ca
AddLanguage cs .cz .cs
AddLanguage da .dk
AddLanguage de .de
AddLanguage el .el
AddLanguage en .en
AddLanguage eo .eo
AddLanguage es .es
AddLanguage et .et
AddLanguage fr .fr
AddLanguage he .he
AddLanguage hr .hr
AddLanguage it .it
AddLanguage ja .ja
AddLanguage ko .ko
AddLanguage ltz .ltz
AddLanguage nl .nl
AddLanguage nn .nn
AddLanguage no .no
AddLanguage pl .po
AddLanguage pt .pt
AddLanguage pt-BR .pt-br
AddLanguage ru .ru
AddLanguage sv .sv
AddLanguage zh-CN .zh-cn
AddLanguage zh-TW .zh-tw
LanguagePriority en ca cs da de el eo es et fr he hr it ja ko ltz nl nn no pl pt pt-BR ru sv zh-CN zh-TW

ForceLanguagePriority Prefer Fallback
AddDefaultCharset UTF-8
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz

AddHandler type-map va
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml


Alias /error/ "/var/www/error/"

<IfModule mod_negotiation.c>
<IfModule mod_include.c>
 <Directory "/var/www/error">
 AllowOverride None
 Options IncludesNoExec
 AddOutputFilter Includes html
 AddHandler type-map var
 Order allow,deny
 Allow from all
 LanguagePriority en es de fr
 ForceLanguagePriority Prefer Fallback
 </Directory>
</IfModule>
</IfModule>

BrowserMatch "Mozilla/2" nokeepalive
BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
BrowserMatch "RealPlayer 4\.0" force-response-1.0
BrowserMatch "Java/1\.0" force-response-1.0
BrowserMatch "JDK/1\.0" force-response-1.0

BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully
BrowserMatch "MS FrontPage" redirect-carefully
BrowserMatch "^WebDrive" redirect-carefully
BrowserMatch "^WebDAVFS/1.[0123]" redirect-carefully
BrowserMatch "^gnome-vfs/1.0" redirect-carefully
BrowserMatch "^XML Spy" redirect-carefully
BrowserMatch "^Dreamweaver-WebDAV-SCM1" redirect-carefully

<Location /status>
 SetHandler server-status
 Order deny,allow
 Deny from all
 Allow from 00.00.00.00
 Allow from 00.00.00.00
</Location>

[trillianjedi]

Lock ups like that in my experience are usually not Apache, but some related process (like a DB engine).

What do you look at when it happens? Output of TOP? Tailing all relevant logfiles ?

[cranky7]

Thanks for the reply.

When it happens we have been unable to see anything abnormal looking at top or the logs. The user's process seems to just sit there and live forever until we manually kill it, or until an hour or two have passed and it eventually dies. The hardest part is that it is so sporadic, it is very hard to track down and catch. Also with thousands of hits a minute, it's especially hard to isolate the one user/occurrence.

Any suggestions on specific tests to try or areas to look?

[Key_Master]

Infinite looping caused by script or rewrite of some sort?

[cranky7]

Hi Key_Master,

Unfortunately I don't think so. The problem occurs on our production site, and does not seem to be correlated to any one page. If there was a scripting problem, we would likely see a significant occurrence of it.

Any thoughts, questions, or advice is much appreciate!

[trillianjedi]

If there was a scripting problem, we would likely see a significant occurrence of it.

Not necessarily - it's just an edge case, and my money is still on a rogue piece of script code (and given not page specific, something common to all like an include file).

If TOP is not revealing a CPU-bound issue, then it's not a tight loop.

It sounds to me like the thread is actually crashing (or blocked on an I/O request), and Apache isn't releasing the "session' (hence you'll have to restart browser and come back a couple of hours later).

I would try and rule out an I/O request block first:-

1. If you're using PHP, try doing a GREP on your includes/ directory(s) for "flock" and see if you have anything doing disk access which might potentially block on a lock request.

2. Do same GREP for "fopen" and see if you have anything which maybe *should* be wrapped in an flock critical section (check if a thread might be crashing as a result of a read and a write on a file by two separate threads).

3. Do a GREP for "exec()" type functions - are you jumping out to native server land on anything? It could be crashing and failing to return, leaving the thread in a blocked state.

If you're not using PHP then replace "fopen" and "flock" with relevant functions from your scripting language.

Then I'd start looking at the possibility that the child thread is crashing:-

1. Add the core dump directive to your httpd.conf. From memory I think you just specify a directory, like "CoreDumpDirectory /apache_dumps".

2. Check your OS to ensure there are no mem limits on core dumps. Sanity check that having this on will not impact server performance too much. Wait for the problem to occur, let it all timeout after a few hours.

3. See if you get a core dump in that directory. If you do, next step is get a stack trace out of it and post it here so we can take a look.

[cranky7]

Thank! I will do this next time it happens and will report back with the status.