Forum Moderators: phranque

Message Too Old, No Replies

FastCGI problems (apache 2.2, php 5.1, suexec)

randomly fails

         

wesleyh84

7:13 pm on Dec 17, 2008 (gmt 0)

10+ Year Member



I have just switched from mod_php to fastcgi + suexec, but now trying to use the ab command times out :


ab -n 10000 -c 100 http://www.example.com/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, [zeustech.net...]
Copyright 2006 The Apache Software Foundation, [apache.org...]

Benchmarking www.example.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
apr_poll: The timeout specified has expired (70007)
Total of 7422 requests completed

Then I get a bunch of emails from LFD


Onderwerp: lfd on server.example.com: Excessive processes running under user example

Time: Wed Dec 17 05:08:48 2008 -0600
Account: example
Process Count: 61 (Not killed)

Process Information:

User:example PID:4515 Run Time:51(secs) Memory:181096(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4516 Run Time:51(secs) Memory:181096(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4517 Run Time:51(secs) Memory:181084(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4518 Run Time:51(secs) Memory:181092(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4523 Run Time:50(secs) Memory:181084(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4524 Run Time:50(secs) Memory:181096(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4525 Run Time:50(secs) Memory:181092(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4526 Run Time:50(secs) Memory:181084(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4527 Run Time:50(secs) Memory:181084(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4528 Run Time:50(secs) Memory:181088(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi
User:example PID:4540 Run Time:50(secs) Memory:181092(kb) exe:/usr/bin/php-cgi cmd:/usr/bin/php-cgi

(and about 50 more such lines)

Also a ton of these emais:


Onderwerp: lfd on server.example.com: Excessive resource usage: example (8269)

Time: Wed Dec 17 05:09:58 2008 -0600
Account: example
Resource: Virtual Memory Size
Exceeded: 176 > 100 (MB)
Executable: /usr/bin/php-cgi
Command Line: /usr/bin/php-cgi
PID: 8269
Killed: No

So what's going on here? Any ideas? I knew fastcgi would use more memory.. but this?

I then made some changes to the wrapper, and removed PHP_FCGI_CHILDREN as some people said that it is bad to use it?

Well, I do get less timeouts now, they happen every few times I try the ab command though. Here is an ab that succeeded:


[root@server ~]# ab -n 10000 -c 100 http://www.example.com/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, [zeustech.net...]
Copyright 2006 The Apache Software Foundation, [apache.org...]

Benchmarking www.example.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Finished 10000 requests

Server Software: Apache
Server Hostname: www.example.com
Server Port: 80

Document Path: /
Document Length: 6724 bytes

Concurrency Level: 100
Time taken for tests: 51.277226 seconds
Complete requests: 10000
Failed requests: 20
(Connect: 0, Length: 20, Exceptions: 0)
Write errors: 0
Non-2xx responses: 20
Total transferred: 69465480 bytes
HTML transferred: 67116200 bytes
Requests per second: 195.02 [#/sec] (mean)
Time per request: 512.772 [ms] (mean)
Time per request: 5.128 [ms] (mean, across all concurrent requests)
Transfer rate: 1322.95 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 7
Processing: 0 489 4464.7 0 51032
Waiting: 0 489 4464.7 0 51032
Total: 0 489 4464.7 0 51032

Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 17
80% 18
90% 32
95% 37
98% 67
99% 13000
100% 51032 (longest request)

See 99 and 100%, why are those taking so long in comparison to the rest?

Is it normal that this stuff is so high and so much during an ab test:


Real memory: 3.86 GB total / 1.70 GB free Swap space: 8 GB total / 7.98 GB free

18024apache866880 kB/usr/sbin/httpd.worker <--- 896MB ?...
17872apache801488 kB/usr/sbin/httpd.worker
17838apache801340 kB/usr/sbin/httpd.worker
2669mysql419436 kB/usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-f ...
18166apache409768 kB/usr/sbin/httpd.worker
18164apache409756 kB/usr/sbin/httpd.worker
3141root240748 kB/usr/bin/python -tt /usr/sbin/yum-updatesd
7528example223900 kB/usr/bin/php-cgi <--- 223MB per php-cgi process? (well is this some sort of virtual memory since obviously it can't be really all that big)
7520example223068 kB/usr/bin/php-cgi
8203example215388 kB/usr/bin/php-cgi
8653example215388 kB/usr/bin/php-cgi
9550example215388 kB/usr/bin/php-cgi
10357example215388 kB/usr/bin/php-cgi
10446example215388 kB/usr/bin/php-cgi
10895example215388 kB/usr/bin/php-cgi
11078example215388 kB/usr/bin/php-cgi
11440example215388 kB/usr/bin/php-cgi
11702example215388 kB/usr/bin/php-cgi
11956example215388 kB/usr/bin/php-cgi
12743example215388 kB/usr/bin/php-cgi
13009example215388 kB/usr/bin/php-cgi
13532example215388 kB/usr/bin/php-cgi
13928example215388 kB/usr/bin/php-cgi
14436example215388 kB/usr/bin/php-cgi
8296example215384 kB/usr/bin/php-cgi
8566example215384 kB/usr/bin/php-cgi
8739example215384 kB/usr/bin/php-cgi
9730example215384 kB/usr/bin/php-cgi
10990example215384 kB/usr/bin/php-cgi
11746example215384 kB/usr/bin/php-cgi
11831example215384 kB/usr/bin/php-cgi
12219example215384 kB/usr/bin/php-cgi
13100example215384 kB/usr/bin/php-cgi
13187example215384 kB/usr/bin/php-cgi
13449example215384 kB/usr/bin/php-cgi
14095example215384 kB/usr/bin/php-cgi
15038example215384 kB/usr/bin/php-cgi
15380example215384 kB/usr/bin/php-cgi
7684example215380 kB/usr/bin/php-cgi
7686example215380 kB/usr/bin/php-cgi
7795example215380 kB/usr/bin/php-cgi
9188example215380 kB/usr/bin/php-cgi
9818example215380 kB/usr/bin/php-cgi
9993example215380 kB/usr/bin/php-cgi
10626example215380 kB/usr/bin/php-cgi
10709example215380 kB/usr/bin/php-cgi
11350example215380 kB/usr/bin/php-cgi
12306example215380 kB/usr/bin/php-cgi
12656example215380 kB/usr/bin/php-cgi
13204example215380 kB/usr/bin/php-cgi
13838example215380 kB/usr/bin/php-cgi
14011example215380 kB/usr/bin/php-cgi
14525example215380 kB/usr/bin/php-cgi
14859example215380 kB/usr/bin/php-cgi
15036example215380 kB/usr/bin/php-cgi
15296example215380 kB/usr/bin/php-cgi
12044example215376 kB/usr/bin/php-cgi
13360example215376 kB/usr/bin/php-cgi
7687example215264 kB/usr/bin/php-cgi
7690example215264 kB/usr/bin/php-cgi
8162example215264 kB/usr/bin/php-cgi
9006example215264 kB/usr/bin/php-cgi
9371example215264 kB/usr/bin/php-cgi
9462example215264 kB/usr/bin/php-cgi
10084example215264 kB/usr/bin/php-cgi
11612example215264 kB/usr/bin/php-cgi
12389example215264 kB/usr/bin/php-cgi
12831example215264 kB/usr/bin/php-cgi
12920example215264 kB/usr/bin/php-cgi
13667example215264 kB/usr/bin/php-cgi
14246example215264 kB/usr/bin/php-cgi
7691example215260 kB/usr/bin/php-cgi
7692example215260 kB/usr/bin/php-cgi
7694example215260 kB/usr/bin/php-cgi
8833example215260 kB/usr/bin/php-cgi
10267example215260 kB/usr/bin/php-cgi
11265example215260 kB/usr/bin/php-cgi
14260example215260 kB/usr/bin/php-cgi
14703example215260 kB/usr/bin/php-cgi
15129example215260 kB/usr/bin/php-cgi
7685example215256 kB/usr/bin/php-cgi
7693example215256 kB/usr/bin/php-cgi
7981example215256 kB/usr/bin/php-cgi
8070example215256 kB/usr/bin/php-cgi
8474example215256 kB/usr/bin/php-cgi
8920example215256 kB/usr/bin/php-cgi
9098example215256 kB/usr/bin/php-cgi
9277example215256 kB/usr/bin/php-cgi
10171example215256 kB/usr/bin/php-cgi
12484example215256 kB/usr/bin/php-cgi
12514example215256 kB/usr/bin/php-cgi
13755example215256 kB/usr/bin/php-cgi
14349example215256 kB/usr/bin/php-cgi
14611example215256 kB/usr/bin/php-cgi
14946example215256 kB/usr/bin/php-cgi
7688example215252 kB/usr/bin/php-cgi
7888example215252 kB/usr/bin/php-cgi
8386example215252 kB/usr/bin/php-cgi
9640example215252 kB/usr/bin/php-cgi
9907example215252 kB/usr/bin/php-cgi
10532example215252 kB/usr/bin/php-cgi
10806example215252 kB/usr/bin/php-cgi
11173example215252 kB/usr/bin/php-cgi
11526example215252 kB/usr/bin/php-cgi
12134example215252 kB/usr/bin/php-cgi
13547example215252 kB/usr/bin/php-cgi
15467example215252 kB/usr/bin/php-cgi
6842apache128268 kB/usr/sbin/httpd.worker
5778root128232 kB/usr/sbin/httpd.worker <-- is this normal (root) ?

These stay loaded even after the ab has finished running. (hours later, still there) This was a server that had 300MB ram in use when using mod_php..

After checking the error log, these are the errors i found:

in /home/example/logs/error_log


[Wed Dec 17 10:29:57 2008] [warn] (104)Connection reset by peer: mod_fcgid: read data from fastcgi server error.
[Wed Dec 17 10:29:57 2008] [error] [client 69.65.x.x] Premature end of script headers: index.php

(multiple times)

in /var/log/httpd/error_log


[Wed Dec 17 10:30:01 2008] [notice] mod_fcgid: call /home/example/public_html/index.php with wrapper /home/example/fcgi-bin/php5.fcgi
[Wed Dec 17 10:30:10 2008] [notice] mod_fcgid: process /home/example/public_html/index.php(7693) exit(normal exit), terminated by calling exit(), return code: 0

(multiple times)


[Wed Dec 17 10:30:10 2008] [warn] mod_fcgid: cleanup zombie process 7693

Here are my config files:

My fcgi wrapper:


#!/bin/sh
PHPRC=$PWD/../etc/php5
export PHPRC
umask 022
SCRIPT_FILENAME=$PATH_TRANSLATED
export SCRIPT_FILENAME
exec /usr/bin/php-cgi

fcgid.conf (apache)


AddHandler fcgid-script .php

# Sane place to put sockets and shared memory file
SocketPath run/mod_fcgid
SharememPath run/fcgid_shm

IdleTimeout 3600
ProcessLifeTime 7200
IPCConnectTimeout 30
IPCCommTimeout 45
MaxRequestsPerProcess 500

Virtualhost:


SuexecUserGroup example example

...

AddHandler fcgid-script .php
FCGIWrapper /home/example/fcgi-bin/php5.fcgi .php

...

Any help is appreciated, thanks.