Possible dead lock [LSAPI application] - Help

#24
Hi @Dolirow ,

You hit the limit somehow, please increase both number from 60 to e.g. 100
Code:
maxConns                60
env                     PHP_LSAPI_CHILDREN=60
Then restart the PHP by killall lsphp
Thanks!!

$ killall lsphp
lsphp: no process found

so I did:
systemctl restart lsws

still logs: No request delivery notification has been received from LSAPI application, possible dead lock.
 
#27
@Cold-Egg
Hi, I updated to 7.3.15 and now appear another type of erros (many of them):
still error: No request delivery notification has been received from LSAPI application, possible dead lock.

Code:
2020-07-24 21:06:38.044501 [NOTICE] sendKillCmdToWatchdog: 'extappkill:982:-3:0'.
2020-07-24 21:06:38.800495 [NOTICE] [438] Cmd from child: [extappkill:982:-3:0]
2020-07-24 21:06:38.800520 [INFO] Failed to get process [982] start time, not running, skip killing.
2020-07-24 21:07:08.631219 [NOTICE] [LocalWorker::workerExec] VHost:_AdminVHost suExec check uid 999 gid 65534 setuidmode 2.
2020-07-24 21:07:08.631235 [NOTICE] [LocalWorker::workerExec] Config[AdminPHP]: suExec uid -1 gid -1 cmd /usr/local/lsws/admin/fcgi-bin/admin_php -c ../conf/php.ini, final uid 999 gid 65534.
2020-07-24 21:07:08.631407 [NOTICE] [AdminPHP] add child process pid: 1016
 
Last edited:

Firsh

New Member
#32
FWIW I think I solved it in my specific case. I'm a WP developer and I believe my refreshes of the Dashboard with "Disable cache" in the developer tools of Chrome might have overwhelmed the server with the lots of little scripts and uncached resources, as it's anything but serving static HTML pages, PHP is polled a lot. I checked on the OLS WebAdmin Console that the aforementioned message was plentiful in the log. After I made the following changes to Server Configuration > External App > lsphp, the notices in the error log changed to simply "[AdminPHP] add child process" followed by "Cmd from child: [extappkil...".

Max Connections: 1000
Code:
PHP_LSAPI_CHILDREN=35
LSAPI_AVOID_FORK=1
PHPRC=/usr/local/lsws/php/php.ini
The fork was a memory value which I have plenty of (It's an Ubuntu VM). The connections and children values were much lower. I believe while the OLS itself is responsive and in it I already bumped the connection counts, it's PHP itself that wasn't spawning enough child processes or hit connection limits and that made my dashboard hang a lot. It's kinda nerve-wracking when you want to work but the server is unresponsive, but you own the server, it's local, and you give it all the resources in the world.
 
#33
Nothing of yours works, the errors will only be fixed when you set the Server configuration "CPU Affinity" from 0 to eg. 2
I have wasted some time tracking it down
~Muchomor
 
#35
"No request delivery notification has been received from LSAPI application, possible dead lock."



Erro logs:
Code:
2020-06-26 12:46:25.005145 [NOTICE] [121.101.104.52:20750#Example] No request delivery notification has been received from LSAPI application, possible dead lock.
2020-06-26 12:46:25.405862 [INFO] [162.157.107.142:16000#Example] Connection idle time too long: 31 while in state: 6 watching for event: 0, close!
2020-06-26 12:46:25.405938 [INFO] [162.158.137.122:16000] Content len: 0, Request line:
GET /index.php?subtopic=houses&show=2163 HTTP/1.1
2020-06-26 12:46:25.405959 [INFO] [162.158.117.122:16000#Example] HttpExtConnector state: 8, request body sent: 0, response body size: -2, response body sent:0, left in buffer: 0, attempts: 0.
2020-06-26 12:46:27.016562 [NOTICE] [172.68.24.261:29042#Example] No request delivery notification has been received from LSAPI application, possible dead lock.
2020-06-26 12:46:27.016654 [NOTICE] [172.68.22.261:29040#Example] No request delivery notification has been received from LSAPI application, possible dead lock.
2020-06-26 12:46:27.045244 [NOTICE] [172.68.24.261:29118#Example] No request delivery notification has been received from LSAPI application, possible dead lock.
2020-06-26 12:46:27.045308 [NOTICE] [172.68.24.261:29136#Example] No request delivery notification has been received from LSAPI application, possible dead lock.
2020-06-26 12:46:27.065838 [NOTICE] [172.68.24.261:29054#Example] No request delivery notification has been received from LSAPI application, possible dead lock.
2020-06-26 12:46:27.069825 [NOTICE] [172.68.24.261:29068#Example] No request delivery notification has been received from LSAPI application, possible dead lock.
2020-06-26 12:46:27.069888 [NOTICE] [172.68.24.261:29104#Example] No request delivery notification has been received from LSAPI application, possible dead lock.
Try checking your zlib setting in php.ini. setting compression on largely solved the deadlock issue for me. See the thread:
OpenLiteSpeed stops working. | OpenLiteSpeed Community and News
 
#36
I've recently had to stop using OLS (and return to Apache2, which does work but slowly) because of this problem. :-(

I accept that it may not be a bug in OLS itself - the root cause is probably more likely to be a bug in a WordPress plugin or a misconfiguration of OLS that causes it to run out of resources. The problem is, there's no clear guidance for isolating the issue, but it has been reported for years and must be damaging uptake of OLS.

For me, the problem is intermittent and none of the suggested configuration tweaks (for example to cpuAffinity or zlib.output_compression) have helped.

Apart from multiple "No request delivery notification has been received from LSAPI application, possible dead lock" errors, I haven't been able to find anything significant in any error logs. I don't see any hung processes when viewed with "top". Switching from memcached to redis seemed to help for a while but the problem returned. Intermittent problems are a bitch. When it happens, it seems to simultaneously affect the back end of multiple sites more that the front end - it doesn't trigger site load speed monitors and I only know about it when admins start complaining that everything runs "like treacle" or they have seen 503 timeouts. The problem can be fixed for a while (a few days) by restarting everything.

It's a production server (small VPS, 4 MB RAM, 2 cores and about 50 sites all running on Virtualmin and Ubuntu 22.04) so there is a limit to the testing that can be done. Before deploying it I did run some load tests to check resource settings, and it seemed reliable (crucially, more reliable than Apache2).

Does anyone have guidance on tweaking the numerous resource settings to avoid this? Does it affect people using the paid LiteSpeed version as well?
 

Cold-Egg

Administrator
#37
Hi,
I saw you mentioned " I don't see any hung processes when viewed with "top" ", is it possible to share the output when the 503 issue happens? For a 4 GB RAM server with 50 sites plus control panel, my guess is there are too many PHP processes running at the same time and the server does not have enough resources to handle them.
 
#38
Thanks for your response. I'm a bit reluctant to knowingly inflict pain on my users, but I'll do it once if we're prepared to gather all necessary information in one go. Which "resources" need to be monitored, exactly?

I should say that the "possible dead lock" messages are much more common than 503 errors, which only happen when the server gets so slow it hits timeouts.

Also, I routinely monitor CPU, RAM, network and disk usage and I don't believe the server is "overloaded" as such. Apache is quite happily serving the same sites as I speak. Before the server was commissioned I ran a whole lot of load tests using loadimpact.com and it's nowhere near the level that caused timeouts in those tests.

Having said that, it's quite clear that there's either some sort of resource issue, or a "dead lock" bug as the error message implies. Hence my question, "which resources need to be monitored?".

I'm happy to give access to the control panel if that would help.
 
Top