Reached max children process - Log into vhost error log instead of stderr

paulozlx · Oct 9, 2019

Hi guys

I noticed OLS is logging every "Reached max children process" kind of error on stderr.log instead of the vhost's own error.log
In another post (https://forum.openlitespeed.org/threads/custom-error-log-for-vhost-not-working.2620/) someone said that stderr always go to stderr.log "
due to it being more of a server level type error".

I' not sure this should be the case with PHP_CHILDREN kind of error. If we have, lets say, 10 sites in the server, there is no way to know which one is complaining about the max children limit, since the log doesn't say anything about it. The only message logged is this:
[1488] Reached max children process limit: XX, extra: 0, current: XX, busy: XX, please increase LSAPI_CHILDREN.

This makes debugging a lot harder than it should be.

Is there someway to log this into vhosts own log?

PS: The same applies to node log. Every error is being logged on stderr.log instead of the vhost/context log. :/

Thanks!

Pong · Oct 10, 2019

There are certain rules to log which one to which by web server by design. stderr.log are more SA of the server, error log would be for end user and Systeam Admin. If you are SA for the server, you should check both error log and stderr.log all the time. For example, you will need to check both when troubelshooting 503 errors. https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:php:503-errors

paulozlx · Oct 10, 2019

I understand that. But I'm asking you to see more a scenario where we have 2000 managed cloud servers, with 1 to 400 sites in each.
Most of those servers are running nginx with php-fpm. On those servers, I can easily spot any problem checking the php-fpm.log, since it clearly say which pool is suffering:

Code:

[07-Oct-2019 11:55:03] WARNING: [pool xxxxxxxxxx.com.br] server reached pm.max_children setting (2), consider raising it
[07-Oct-2019 11:56:02] WARNING: [pool yyyyyyyyy.com.br] server reached pm.max_children setting (2), consider raising it
[07-Oct-2019 11:56:04] WARNING: [pool zzzzzzzzzzzz.configr.cloud] server reached pm.max_children setting (2), consider raising it
[07-Oct-2019 11:56:39] WARNING: [pool aaaaaaa.configr.cloud] server reached pm.max_children setting (2), consider raising it

But with OLS we only have this information:

Code:

[1488] Reached max children process limit: XX, extra: 0, current: XX, busy: XX, please increase LSAPI_CHILDREN.

What 1488 stands for? How can I easily know which vhost is reaching max children?

This is a relevant information we still don't have with OLS, and it would sure make my life and of other sysadmins lifes easier.
Months ago I wrote a simple bash script to parse php-fpm.log and alert when our client when some sites is reaching max children so he can tweak/buy more resources. With OLS we can't give this kind of information, just because this specific log is irrelevant and doesn't contain any useful information.

It seems OLS is not ready to be largely used in shared hosting yet

Anyway, I kindly ask you to really consider giving more useful information on stderr.log so everyone can better understand and manage the server

PS: Sorry if I'm bothering you, I just want to share my long experience and thoughts with you since I believe OLS has the potential to be the next more used webserver and its really cool to work with it and be one of the few on Brazil hosting thousands of websites with it

David · Oct 10, 2019

Can you add a setting in the php ENV?
such as `LSAPI_STDERR_LOG=[The Vhost Directory]/stderr.log`
For example my test setting

Code:

extprocessor lsphp {
  type                    lsapi
  address                 uds://tmp/lshttpd/lsphp.sock
  maxConns                35
  env                     PHP_LSAPI_CHILDREN=35
  env                      LSAPI_STDERR_LOG=/tmp/stderr.log
  initTimeout             600
  retryTimeout            0

David · Oct 10, 2019

Are you set each VHost has its own lsphp or all vhost share a same lsphp?

paulozlx · Oct 10, 2019

Hi David. Great suggestion as always

Someday I will write a compact guide about all the things we can accomplish with env variables and .htaccess ENV too

But it doesn't seem to be working as expected:
root@ip-95-216-157-207:/srv/zlx.com.br/log/php# ls -lah /tmp/stderr.log
-rw-r--r-- 1 zlx www-data 0 Oct 10 13:10 /tmp/stderr.log

I missconfigured some extensions on PHP to generate a lot of output on stderr, and they are all going to the default stderr.log, the new one stays empty. I even tried some absurd configurations and setting STDERR_LOG to /dev/null as suggested on https://www.litespeedtech.com/suppo..._wiki:troubleshooting:stderr.log-fill-up-disk
Still the log is written on default stderr.

And yes, we set one vhost per site. This is the one being tested:

Code:

extprocessor xxxxxxxxxxx.configr.cloud {
  type                    lsapi
  address                 UDS://srv/xxxxxxxxxxx.configr.cloud/etc/php/lsws.sock
  maxConns                2
  env                     LSAPI_STDERR_LOG=/dev/null
  env                     LSAPI_CHILDREN=1
#  env                     LSAPI_STDERR_LOG=/srv/xxxxxxxxxxx.configr.cloud/log/php/lsws-stderr.log
  initTimeout             3000
  retryTimeout            0
  persistConn             1
  respBuffer              0
  autoStart               2
  path                    /usr/local/lsws/lsphp73/bin/lsphp
  backlog                 100
  instances               2
  extUser                 zlx
  extGroup                www-data
  umask                   022
  runOnStartUp            0
  priority                0
  extMaxIdleTime          10
}

David · Oct 10, 2019

OK, I will do more testing about it.
BTW, your `LSAPI_CHILDREN=1` seems is too low.

paulozlx · Oct 10, 2019

Yeah, I know. I was trying to simulate the "Max children" error.

But believe me, a lot of our clients DEMAND to limit the php process to 2~4, so they can put like 50 wordpress sites on a 1gb-ram server haha

David · Oct 10, 2019

Good to know.
Will update you later.

David · Oct 10, 2019

I tested with vhost config

Code:

phpIniOverride  {
php_admin_value log_errors on
}

and that VHost can log its own php errors.

David · Oct 10, 2019

So the total testing vhost conf is like this

Code:

phpIniOverride  {
php_admin_value log_errors on
}

extprocessor lsphp {
  type                    lsapi
  address                 uds://tmp/lshttpd/lsphp1.sock
  maxConns                1
  env                     PHP_LSAPI_CHILDREN=1
  env                     LSAPI_STDERR_LOG=/tmp/stderr1.log
  initTimeout             600
  retryTimeout            0
  persistConn             1
  respBuffer              0
  autoStart               1
  path                    fcgi-bin/lsphp
  backlog                 100
  instances               1
  priority                0
  memSoftLimit            2047M
  memHardLimit            2047M
  procSoftLimit           400
  procHardLimit           500
}

paulozlx · Oct 10, 2019

Hmm, I tried that with no luck. What OLS version are you using?
I have loaded some extensions twice, so they can produce a PHP Warning, and all warnings are going to /usr/local/lsws/logs/stderr.log instead of /tmp/stderr1.log :/
I'm using LiteSpeed/1.5.6 Open

David · Oct 10, 2019

Are you use this
```
env LSAPI_STDERR_LOG=/dev/null

``` in your conf?

Which lsphp version you are using?

I just tested with a 1.4 and lsphp 5.3.6.

paulozlx · Oct 23, 2019

Hey! So sorry for the long delay to reply here!

Just want to let you know that it is finally working. I don't know if I did something wrong before or was the version I was using (now I'm using 1.5.7), but I can now see which site is reaching max children limit

root@ip-95-216-157-xxx:/srv# tail -f /srv/xxxxxx.com.br/log/lsws/stderr.log
2019-10-23 11:10:46.301759 [28291] Reached max children process limit: 2, extra: 0, current: 2, busy: 2, please increase LSAPI_CHILDREN.

This will help us to continue moving all our nginx+phpfpm users to OLS, finally

As always, thanks for your help and good work ;D

David · Oct 23, 2019

Ok. Good to know and good luck.

paulozlx · Oct 28, 2019

Hey David, how are you?

I'm hoping you can help me out again in a similar problem. I'm trying to log NODE stderr to a different file too.
I tried to do this:

Code:

context / {
  type                    appserver
  env                     LSAPI_STDERR_LOG=/srv/xxxxxxxxxxxx.configr.cloud/log/lsws/stderr.log
  include                 /srv/xxxxxxxxxxxx.configr.cloud/.env
  location                /srv/xxxxxxxxxxxx.configr.cloud/www
  binPath                 /var/www/.virtualenvs/xxxxxxxxxxxx.configr.cloud/10.13.0/bin/node
  appType                 node
  startupFile             index.js
  addDefaultCharset       off
}

But it didn't work. It keeps logging things to default stderr.log. I tried to use LSNODE_CONSOLE_LOG as an env to log things to a specific file, but also didn't work.
Do you know a better way to handle this with node?

David · Oct 28, 2019

I am not so good at using nodejs, you may create a new topic and wait others such coldegg to help you.

w3services_net · Dec 4, 2020

My setting is --

[root@cloudbox logs]# cat /usr/local/lsws/conf/httpd_config.conf | grep LSAPI_CHILDREN
env PHP_LSAPI_CHILDREN=150

Still I am getting error like -

2020-12-04 12:29:09.866 [STDERR] [3372] Reached max children process limit: 10, extra: 3, current: 13, busy: 13, please increase LSAPI_CHILDREN.
2020-12-04 12:29:09.966 [STDERR] [3372] Reached max children process limit: 10, extra: 3, current: 13, busy: 13, please increase LSAPI_CHILDREN.
2020-12-04 12:29:10.066 [STDERR] [3372] Reached max children process limit: 10, extra: 3, current: 13, busy: 13, please increase LSAPI_CHILDREN.

How to fix this? From where it getting max children 10?

gilles · Dec 5, 2020

You may have to do the same thing on your virtual hosts if you have defined external apps there. Also make sure that the 'Max Connections' parameter is also set to the same value (150 in your case).

logiccmpl · Aug 25, 2021

@gilles

Please how can I set the max connection

Reached max children process - Log into vhost error log instead of stderr

Member

Administrator

Member

Active Member

Active Member

Member

Active Member

Member

Active Member

Active Member

Active Member

Member

Active Member

Member

Active Member

Member

Active Member

Member

Active Member

New Member