Need advice on OLS config (520 errors + "please increase LSAPI_CHILDREN")

[gilles]

I am running a few Wordpress websites with Cyperpanel (v2.0) and OLS (v1.7.5) on an Ubuntu 20.04 server (4GB, 2 vCPUs) at DigitalOcean. There are also a couple of nodeJS apps running on the same server. I just recently set up all the sites. I am the one generating most of the traffic right now doing testing. I am using LSCache on the two main sites. I am not expecting a lot of traffic. I want the sites to be reliable and fast(-enough) loading.

I get 520 errors from Cloudflare if I keep hitting refresh on the same website. I also noticed the following errors in the stderr.log:

2020-10-31 02:33:11.939 [STDERR] [581269] Reached max children process limit: 10, extra: 3, current: 13, busy: 13, please increase LSAPI_CHILDREN.
2020-10-31 02:33:12.205 [STDERR] [581269] Reached max children process limit: 10, extra: 3, current: 13, busy: 13, please increase LSAPI_CHILDREN.
[581269] Reached max children process limit: 10, extra: 3, current: 13, busy: 13, please increase LSAPI_CHILDREN.
[581269] Reached max children process limit: 10, extra: 3, current: 13, busy: 12, please increase LSAPI_CHILDREN.
[581269] Reached max children process limit: 10, extra: 3, current: 13, busy: 10, please increase LSAPI_CHILDREN.
2020-10-31 02:33:12.207 [STDERR] [581269] Reached max children process limit: 10, extra: 3, current: 13, busy: 9, please increase LSAPI_CHILDREN.
2020-10-31 02:33:12.228 [STDERR] [581269] Reached max children process limit: 10, extra: 3, current: 13, busy: 12, please increase LSAPI_CHILDREN.

I've also seen the following error:

No request delivery notification has been received from LSAPI application, possible dead lock.

I'm guessing I need to tweak some of the server or vhost settings. I'm also wondering if I should switch the server to 8GB/4 vCPUs.
I am reading documentation but it is more descriptive than directive (and I understand there are a lot of possible scenarios).

I would appreciate advice on:
1) Whether the problem is server related or vhost related? how do I know? which settings should I tweak first?
2) Whether I should switch to a 8GB/4 vCPUs server config.
3) Whether I should look into implementing cgroups manually (Cyberpanel only supports containerization on CentOS for now), or if that won't help my current issues.

 

[Cold-Egg]

Hi,
I guess this post could help, https://forums.cyberpanel.net/discussion/1263/increase-lsapi-children.
Please try it and do more test before you decide to upgrade 4CPU/8GB

 

[gilles]

Thanks. I don't see a PHP SuEXEC Max Connections parameter in OpenLiteSpeed. Does it exist only for LiteSpeed Web Server?

What I see for OLS is:
1) A 'Max Connections' parameter for the server lsphp external app.
2) A PHP_LSAPI_CHILDREN parameter for server the lsphp external app.
3) A 'Max Connections' parameter for the vhost lsphp external app.
4) A LSAPI_CHILDREN parameter for the vhost lsphp external app.

I increased #3 and #4 from 10 to 35.
#1 is set to 35 and #2 to 50, wondering if they should match. Can't remember if I set them this way or if this was the default Cyperpanel config.

Things are better but I still get 520 errors every so often, even when running gtmetrix. I've also seen 525 errors.

The only errors on the server side are related to suspended vhosts and mod_security (because I installed it manually and not through Cyberpanel).
On the vhost side, I have a few errors but they seem unrelated:

2020-10-31 01:02:25.376277 [ERROR] [172.68.132.252:17994#mydomain.com] [modcompress---] compressbuf in 16384, return -1 (written 0, flag in 0)
2020-10-31 01:02:25.502536 [ERROR] [172.68.143.65:38438#mydomain.com] [modcompress---] compressbuf in 16384, return -1 (written 0, flag in 0)
2020-10-31 15:07:23.428058 [ERROR] [162.158.75.33:30950-3#mydomain.com] [Module:Cache] createEntry failed.
2020-11-01 01:31:40.507517 [ERROR] [172.69.22.109:54746#mydomain.com] [Module:Cache] createEntry failed.
2020-11-01 11:01:05.002770 [ERROR] [108.162.219.241:29842#mydomain.com] [Module:Cache] createEntry failed.

I will try to set the log level to DEBUG next. Let me know if phpSuExecMaxConn is valid for OLS and if so, where I should set.

Thank you

 

[gilles]

Here are some errors I see in the server error log:

2020-11-01 22:23:36.869198 [NOTICE] [<client IP>:61118:HTTP2-107#_AdminVHost] [STDERR] PHP Warning: socket_connect(): Host lookup failed [-10003]: Unknown server error in /usr/local/lsws/admin/html.open/lib/ControllerBase.php on line 506
2020-11-01 22:23:36.869240 [NOTICE] [<client IP>:61118:HTTP2-107#_AdminVHost] [STDERR] cmd restart failed to connect to server! socket_connect() failed: Unknown server error
2020-11-01 22:24:17.446416 [NOTICE] [<client IP>:61118:HTTP2-185#_AdminVHost] [STDERR] PHP Warning: socket_connect(): Host lookup failed [-10003]: Unknown server error in /usr/local/lsws/admin/html.open/lib/ControllerBase.php on line 506
2020-11-01 22:24:17.446458 [NOTICE] [<client IP>:61118:HTTP2-185#_AdminVHost] [STDERR] cmd restart failed to connect to server! socket_connect() failed: Unknown server error

 

[Cold-Egg]

Hi,

You can ignore phpSuExecMaxConn part for OLS
Please bypass cloudflare for a while and see if you are able to see any issue
You mentioned it's nodeJS apps , so increasing PHP might be no help unless you have some apps use PHP.

 

[gilles]

Correction. The problem are with the standard Wordpress websites. I have two nodejs apps running on the side through a reverse proxy but they are fine.

I am seeing this too:

2020-11-01 23:09:59.899039 [DEBUG] [Module:Cache]createEntry quit due to internal error.
...
2020-11-01 23:10:00.602209 [DEBUG] [SSL: 0x3159138] checkError returned 2, first error: error:00000000:invalid library (0):OPENSSL_internal:invalid library (0), last error: error:00000000:invalid library (0):OPENSSL_internal:invalid library (0)

Seems like I have at least an issue with the openSSL library. Let me see if I can figure this one out.

 

[gilles]

A new error:

2020-11-01 23:41:31.081366 [DEBUG] readyCompressed() compress gzip error /tmp/lshttpd/swap/f/5/2ce22abfa9116b9b6d7f273bcf516d.lsz or file size not suitable for gzip.

 

[gilles]

Hmm the openSSL error seems like an intermediate step OLS goes through:

2020-11-01 23:56:56.228523 [DEBUG] [SSL: 0x1b81288] SSL_accept SSL_do_handshake rc: -1
2020-11-01 23:56:56.228529 [DEBUG] [SSL: 0x1b81288] checkError returned 2, first error: error:00000000:invalid library (0):OPENSSL_internal:invalid library (0), last error: error:00000000:invalid library (0):OPENSSL_internal:invalid library (0)
2020-11-01 23:56:56.228536 [DEBUG] [108.162.216.71:47784] setSSLAgain(), CR, wantRead: 0, wantWrite: 0, allowWrite: 1, allowRead: 1, m_ssl.wantRead: 1, m_ssl.wantWrite: 0, m_ssl.lastRead: 0, m_ssl.lastWrite: 0
2020-11-01 23:56:56.228541 [DEBUG] [108.162.216.71:47784] setSSLAgain(), SW, wantRead: 0, wantWrite: 0, allowWrite: 1, allowRead: 1, m_ssl.wantRead: 1, m_ssl.wantWrite: 0, m_ssl.lastRead: 0, m_ssl.lastWrite: 0
2020-11-01 23:56:56.228546 [DEBUG] [*:443] 1 connections accepted!

That seems unlikely to be my problem.

 

[gilles]

All the errors went away once I disabled the LSCache plugin. But obviously that's not what I want.

I posted in another thread that OLS is serving public cached pages to logged-in users. There seems to be something very broken and I am unfortunately disappointed so far. I've run into all these issues trying to get OLS and LSCache to work. I get no errors and better load times (on gtmetrix) using OLS and WP Super Cache.

I guess I'm not sure whether the issue is with OLS, LSCache, or both. But it seems it's partly with OLS as OLS served the publicly cached page to the private session somehow.