Reached max children process - Log into vhost error log instead of stderr

#1
Hi guys :)

I noticed OLS is logging every "Reached max children process" kind of error on stderr.log instead of the vhost's own error.log
In another post (https://forum.openlitespeed.org/threads/custom-error-log-for-vhost-not-working.2620/) someone said that stderr always go to stderr.log "
due to it being more of a server level type error".

I' not sure this should be the case with PHP_CHILDREN kind of error. If we have, lets say, 10 sites in the server, there is no way to know which one is complaining about the max children limit, since the log doesn't say anything about it. The only message logged is this:
[1488] Reached max children process limit: XX, extra: 0, current: XX, busy: XX, please increase LSAPI_CHILDREN.

This makes debugging a lot harder than it should be.

Is there someway to log this into vhosts own log?

PS: The same applies to node log. Every error is being logged on stderr.log instead of the vhost/context log. :/

Thanks!
 
#3
I understand that. But I'm asking you to see more a scenario where we have 2000 managed cloud servers, with 1 to 400 sites in each.
Most of those servers are running nginx with php-fpm. On those servers, I can easily spot any problem checking the php-fpm.log, since it clearly say which pool is suffering:

Code:
[07-Oct-2019 11:55:03] WARNING: [pool xxxxxxxxxx.com.br] server reached pm.max_children setting (2), consider raising it
[07-Oct-2019 11:56:02] WARNING: [pool yyyyyyyyy.com.br] server reached pm.max_children setting (2), consider raising it
[07-Oct-2019 11:56:04] WARNING: [pool zzzzzzzzzzzz.configr.cloud] server reached pm.max_children setting (2), consider raising it
[07-Oct-2019 11:56:39] WARNING: [pool aaaaaaa.configr.cloud] server reached pm.max_children setting (2), consider raising it
But with OLS we only have this information:
Code:
[1488] Reached max children process limit: XX, extra: 0, current: XX, busy: XX, please increase LSAPI_CHILDREN.
What 1488 stands for? How can I easily know which vhost is reaching max children?

This is a relevant information we still don't have with OLS, and it would sure make my life and of other sysadmins lifes easier.
Months ago I wrote a simple bash script to parse php-fpm.log and alert when our client when some sites is reaching max children so he can tweak/buy more resources. With OLS we can't give this kind of information, just because this specific log is irrelevant and doesn't contain any useful information.

It seems OLS is not ready to be largely used in shared hosting yet :(

Anyway, I kindly ask you to really consider giving more useful information on stderr.log so everyone can better understand and manage the server

PS: Sorry if I'm bothering you, I just want to share my long experience and thoughts with you since I believe OLS has the potential to be the next more used webserver and its really cool to work with it and be one of the few on Brazil hosting thousands of websites with it :D
 

David

Active Member
#4
Can you add a setting in the php ENV?
such as `LSAPI_STDERR_LOG=[The Vhost Directory]/stderr.log`
For example my test setting
Code:
extprocessor lsphp {
  type                    lsapi
  address                 uds://tmp/lshttpd/lsphp.sock
  maxConns                35
  env                     PHP_LSAPI_CHILDREN=35
  env                      LSAPI_STDERR_LOG=/tmp/stderr.log
  initTimeout             600
  retryTimeout            0
 
Last edited:
#6
Hi David. Great suggestion as always :)
Someday I will write a compact guide about all the things we can accomplish with env variables and .htaccess ENV too

But it doesn't seem to be working as expected:
root@ip-95-216-157-207:/srv/zlx.com.br/log/php# ls -lah /tmp/stderr.log
-rw-r--r-- 1 zlx www-data 0 Oct 10 13:10 /tmp/stderr.log

I missconfigured some extensions on PHP to generate a lot of output on stderr, and they are all going to the default stderr.log, the new one stays empty. I even tried some absurd configurations and setting STDERR_LOG to /dev/null as suggested on https://www.litespeedtech.com/suppo..._wiki:troubleshooting:stderr.log-fill-up-disk
Still the log is written on default stderr.

And yes, we set one vhost per site. This is the one being tested:
Code:
extprocessor xxxxxxxxxxx.configr.cloud {
  type                    lsapi
  address                 UDS://srv/xxxxxxxxxxx.configr.cloud/etc/php/lsws.sock
  maxConns                2
  env                     LSAPI_STDERR_LOG=/dev/null
  env                     LSAPI_CHILDREN=1
#  env                     LSAPI_STDERR_LOG=/srv/xxxxxxxxxxx.configr.cloud/log/php/lsws-stderr.log
  initTimeout             3000
  retryTimeout            0
  persistConn             1
  respBuffer              0
  autoStart               2
  path                    /usr/local/lsws/lsphp73/bin/lsphp
  backlog                 100
  instances               2
  extUser                 zlx
  extGroup                www-data
  umask                   022
  runOnStartUp            0
  priority                0
  extMaxIdleTime          10
}
 
#8
Yeah, I know. I was trying to simulate the "Max children" error. :)

But believe me, a lot of our clients DEMAND to limit the php process to 2~4, so they can put like 50 wordpress sites on a 1gb-ram server haha
 

David

Active Member
#11
So the total testing vhost conf is like this
Code:
phpIniOverride  {
php_admin_value log_errors on
}

extprocessor lsphp {
  type                    lsapi
  address                 uds://tmp/lshttpd/lsphp1.sock
  maxConns                1
  env                     PHP_LSAPI_CHILDREN=1
  env                     LSAPI_STDERR_LOG=/tmp/stderr1.log
  initTimeout             600
  retryTimeout            0
  persistConn             1
  respBuffer              0
  autoStart               1
  path                    fcgi-bin/lsphp
  backlog                 100
  instances               1
  priority                0
  memSoftLimit            2047M
  memHardLimit            2047M
  procSoftLimit           400
  procHardLimit           500
}
 
#12
Hmm, I tried that with no luck. What OLS version are you using?
I have loaded some extensions twice, so they can produce a PHP Warning, and all warnings are going to /usr/local/lsws/logs/stderr.log instead of /tmp/stderr1.log :/
I'm using LiteSpeed/1.5.6 Open
 
#14
Hey! So sorry for the long delay to reply here!

Just want to let you know that it is finally working. I don't know if I did something wrong before or was the version I was using (now I'm using 1.5.7), but I can now see which site is reaching max children limit :)

root@ip-95-216-157-xxx:/srv# tail -f /srv/xxxxxx.com.br/log/lsws/stderr.log
2019-10-23 11:10:46.301759 [28291] Reached max children process limit: 2, extra: 0, current: 2, busy: 2, please increase LSAPI_CHILDREN.

This will help us to continue moving all our nginx+phpfpm users to OLS, finally :D

As always, thanks for your help and good work ;D
 
#16
Hey David, how are you?

I'm hoping you can help me out again in a similar problem. I'm trying to log NODE stderr to a different file too.
I tried to do this:
Code:
context / {
  type                    appserver
  env                     LSAPI_STDERR_LOG=/srv/xxxxxxxxxxxx.configr.cloud/log/lsws/stderr.log
  include                 /srv/xxxxxxxxxxxx.configr.cloud/.env
  location                /srv/xxxxxxxxxxxx.configr.cloud/www
  binPath                 /var/www/.virtualenvs/xxxxxxxxxxxx.configr.cloud/10.13.0/bin/node
  appType                 node
  startupFile             index.js
  addDefaultCharset       off
}
But it didn't work. It keeps logging things to default stderr.log. I tried to use LSNODE_CONSOLE_LOG as an env to log things to a specific file, but also didn't work.
Do you know a better way to handle this with node?
 
#18
My setting is --

[root@cloudbox logs]# cat /usr/local/lsws/conf/httpd_config.conf | grep LSAPI_CHILDREN
env PHP_LSAPI_CHILDREN=150


Still I am getting error like -

2020-12-04 12:29:09.866 [STDERR] [3372] Reached max children process limit: 10, extra: 3, current: 13, busy: 13, please increase LSAPI_CHILDREN.
2020-12-04 12:29:09.966 [STDERR] [3372] Reached max children process limit: 10, extra: 3, current: 13, busy: 13, please increase LSAPI_CHILDREN.
2020-12-04 12:29:10.066 [STDERR] [3372] Reached max children process limit: 10, extra: 3, current: 13, busy: 13, please increase LSAPI_CHILDREN.

How to fix this? From where it getting max children 10?
 

gilles

Active Member
#19
You may have to do the same thing on your virtual hosts if you have defined external apps there. Also make sure that the 'Max Connections' parameter is also set to the same value (150 in your case).
 
Top