warning in PHP-FPM log

[Megunticook]

I'm having a little trouble getting rid of warnings in my PHP log on a dedicated virtual web server on AWS. Here's an example from yesterday:

[25-Nov-2019 06:14:28] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 2 idle, and 17 total children
[25-Nov-2019 06:30:16] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 17 total children
[25-Nov-2019 06:30:17] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 21 total children
[25-Nov-2019 06:30:18] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 3 idle, and 25 total children

The server has 4GB RAM, 2 CPUs. I've been adjusting the values in my PHP-FPM configuration file, www.conf, but still getting these warnings. Here's what I currently have:

pm.max_children = 40
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 10

Earlier I tried increasing pm.max_spare_servers from 8 to 12, but memory use increased from running 20-25% to 60% even during times when there was very little traffic on the site, and I still got the occasional warning in the PHP log (same as above). I dropped max_spare_servers from 12 to 10, which eased up on the memory use, but getting the log warnings still (not every day, but most days--and even at odd times when use is low).

I ran ps -ylC php-fpm7.2 --sort:rss to get a sense of how much memory a typical PHP-FPM process was using:

S  UID  PID PPID C PRI NI  RSS  SZ WCHAN TTY     TIME CMD
S   0  825   1 0 80  0 12300 113501 -   ?    00:00:07 php-fpm7.2
S  111 19966  825 0 80  0 121856 138846 -  ?    00:00:24 php-fpm7.2
S  111 19967  825 0 80  0 129808 140786 -  ?    00:00:25 php-fpm7.2
S  111 19965  825 0 80  0 131740 141210 -  ?    00:00:26 php-fpm7.2
S  111 15265  825 0 80  0 169792 149971 -  ?    00:03:18 php-fpm7.2
S  111 15262  825 1 80  0 171884 149019 -  ?    00:09:42 php-fpm7.2
S  111 14903  825 0 80  0 177012 149796 -  ?    00:03:51 php-fpm7.2
S  111 14641  825 0 80  0 184656 150511 -  ?    00:04:07 php-fpm7.2
S  111 14904  825 0 80  0 185560 151905 -  ?    00:04:03 php-fpm7.2
S  111 15263  825 0 80  0 186804 153696 -  ?    00:03:55 php-fpm7.2
S  111 14902  825 0 80  0 193004 153564 -  ?    00:04:22 php-fpm7.2

The values in the RSS column, indicating memory used by each process, seemed surprisingly high.

I also ran ps --no-headers -o "rss,cmd" -C php-fpm7.2 | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"Mb") }' to compare and got a result of 154 MB average per process. That seems very high, doesn't it?

What do you make of all this and what do you recommend I do to resolve the warnings appearing in the PHP log?

Thanks.

[robzilla]

120-200MB may seem like a lot, but it's quite common for sites running on a bloated CMS like Wordpress. It's also not actually what the processes are using, since much of it is memory shared among all PHP-FPM processes. And note that PHP-FPM using 60% of memory is not necessarily a bad thing... unless there are other processes that need it more.

There are many (complementary) ways to approach this:
- decrease the amount of time PHP spends working on requests, e.g. through caching (make sure an opcode cache is installed);
- rewrite your scripts to use memory and/or CPU-time more efficiently (if you're on a CMS like Wordpress, look into caching plug-ins);
- decrease the number of illegitimate requests (indexers, scrapers and other bots) to reduce load;
- bypass PHP altogether where possible, e.g. through full-page caching where possible (FastCGI cache);
- increase the number of available workers, which may require additional memory (i.e. upgrade your VM).

Hope that helps.

[Megunticook]

Thanks much for your reply.

I've been planning to install opcache, will put that on a fast-track. Am using the W3TC caching plugin with WordPress currently, but only have memcached installed.

What's the best way to block "illegitimate" requests? Don't want to block search engine bots, since this is an ecommerce site so search visibility is key.

When you say increase available workers, in the parlance of my PHP-FPM config do you mean increase pm.max_children?

Thanks again.

[robzilla]

Installing an opcode cache is one of the easiest and quickest things you can do to improve performance and reduce load. I can't remember if one is automatically included with PHP7 nowadays.

As for blocking bots, keyplyr has a good summary of blocking methods [webmasterworld.com] you could use. I'd start with the low-hanging fruit like blocking empty user agent strings and http/1.0 requests, but ideally you would also analyze your access logs to determine priorities here. Maybe there's a particular IP or bot that's hammering your server all the time and blocking it reduces your load issues.

When you say increase available workers, in the parlance of my PHP-FPM config do you mean increase pm.max_children?

Yes. You'll want to be careful with this, of course, as you don't want to go into heavy swapping or crash the server because you're running out of memory.

The message "spawning 32 children, there are 3 idle, and 25 total children" is a bit odd, isn't it? How can you spawn 32 children when the max is set to 40 and you already have 25. Perhaps I'm misreading it. I usually set PHP-FPM to a static number of workers as I don't like the overhead of dynamic worker allocation.

W3TC is a pretty good plug-in and ought to reduce load quite a bit. If your content is static enough, you could consider doing some FastCGI caching with nginx, but it can be difficult to get right.

You could also see if there are any particularly long-running scripts like cronjobs or slow pages hogging these workers. Also make sure that requests for static resources such as images and stylesheets are not passed through PHP-FPM but served directly by the web server (this is usually the case but you never know).

[Megunticook]

Thanks for your advice. W3TC allows you to choose between APC, eAccelerator, or XCache for Opcode. Which of those do you think is the best choice? I'm running PHP 7.2. That'll be my first step here.

I'll check out keyplyr's summary on blocking methods, thanks for that link.

I switched from WPSuperCache to W3TC last summer as it seemed like it had a better setup for using CloudFront CDN and offloading static content to S3, which I've done. I think that's lightened the load on my server significantly.

But obviously still need to dial in my settings to get optimal performance. Thanks again.

[robzilla]

I didn't know there's an opcode cache option in W3TC. Ideally you would enable the opcode cache via your php.ini file (opcache.enable=1).

Maybe also consider installing a monitoring tool like Munin so you can see how your server is holding up throughout the days, weeks, months, etc. It's a good way to tell if your changes have the desired effect.

[Megunticook]

Just a quick follow-up: the php error log is now clean, no errors or warnings. My current PHP-FPM configuration in www.conf is:

pm.max_children = 30
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 11

Terrific. However, I think I still have some work to do. In my NGINX error log, I'm seeing things like this:

2019/12/08 07:39:42 [warn] 20920#20920: *105668 an upstream response is buffered to a temporary file /var/lib/nginx/fastcgi/8/15/0000000158 while reading upstream, client: 172.31.25.241, server: mydomainname.com, request: "GET /product-category/hand-knitting-patterns/page/2/?add_to_wishlist=14314 HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.2-fpm.sock:", host: "mydomainname.com"

Did some Googling on this but I'm not 100% clear on whether this is just expected normal nginx behavior or whether I need to adjust my NGINX configuration.

Thoughts? Thanks.

[robzilla]

It's expected when the FastCGI memory buffer is full, so nginx then writes to a file instead. It's a warning because disk is much slower than RAM. Not a terrible thing, but if you see many of these warnings you might want to increase the buffer size to something that better fits the average response size. You can probably find some tutorials on how to tune those settings.

[Megunticook]

Got it, thank. I'll do some research on that. I did a quick search for the fastcgi_buffers setting and didn't see it. Do you know which config file this is in? Thanks again.

[robzilla]

You can set fastcgi_buffer_size [nginx.org] and fastcgi_buffers in the context of a http{}, server{} or location{} block. So unless you want to customize the buffers to each site, the easiest would be in the http block in nginx.conf.