Possible dead lock [LSAPI application] - Help
- Thread starter Dolirow
- Start date
@Cold-Egg
Hi, I updated to 7.3.15 and now appear another type of erros (many of them):
still error: No request delivery notification has been received from LSAPI application, possible dead lock.
Hi, I updated to 7.3.15 and now appear another type of erros (many of them):
still error: No request delivery notification has been received from LSAPI application, possible dead lock.
Code:
2020-07-24 21:06:38.044501 [NOTICE] sendKillCmdToWatchdog: 'extappkill:982:-3:0'.
2020-07-24 21:06:38.800495 [NOTICE] [438] Cmd from child: [extappkill:982:-3:0]
2020-07-24 21:06:38.800520 [INFO] Failed to get process [982] start time, not running, skip killing.
2020-07-24 21:07:08.631219 [NOTICE] [LocalWorker::workerExec] VHost:_AdminVHost suExec check uid 999 gid 65534 setuidmode 2.
2020-07-24 21:07:08.631235 [NOTICE] [LocalWorker::workerExec] Config[AdminPHP]: suExec uid -1 gid -1 cmd /usr/local/lsws/admin/fcgi-bin/admin_php -c ../conf/php.ini, final uid 999 gid 65534.
2020-07-24 21:07:08.631407 [NOTICE] [AdminPHP] add child process pid: 1016
Last edited:
Hi,
Please check if your app support PHP73. If there's no PHP error, please raise a ticket with us support@litespeedtech.com
Best,
Eric
Please check if your app support PHP73. If there's no PHP error, please raise a ticket with us support@litespeedtech.com
Best,
Eric
For the PHP connection timeout, please try following this guide https://docs.litespeedtech.com/cp/cpanel/long-run-script/#easiest-solution to make PHP Without Timeout.
FWIW I think I solved it in my specific case. I'm a WP developer and I believe my refreshes of the Dashboard with "Disable cache" in the developer tools of Chrome might have overwhelmed the server with the lots of little scripts and uncached resources, as it's anything but serving static HTML pages, PHP is polled a lot. I checked on the OLS WebAdmin Console that the aforementioned message was plentiful in the log. After I made the following changes to Server Configuration > External App > lsphp, the notices in the error log changed to simply "[AdminPHP] add child process" followed by "Cmd from child: [extappkil...".
Max Connections: 1000
The fork was a memory value which I have plenty of (It's an Ubuntu VM). The connections and children values were much lower. I believe while the OLS itself is responsive and in it I already bumped the connection counts, it's PHP itself that wasn't spawning enough child processes or hit connection limits and that made my dashboard hang a lot. It's kinda nerve-wracking when you want to work but the server is unresponsive, but you own the server, it's local, and you give it all the resources in the world.
Max Connections: 1000
Code:
PHP_LSAPI_CHILDREN=35
LSAPI_AVOID_FORK=1
PHPRC=/usr/local/lsws/php/php.iniTry checking your zlib setting in php.ini. setting compression on largely solved the deadlock issue for me. See the thread:
OpenLiteSpeed stops working. | OpenLiteSpeed Community and News
OpenLiteSpeed stops working. | OpenLiteSpeed Community and News
I've recently had to stop using OLS (and return to Apache2, which does work but slowly) because of this problem. :-(
I accept that it may not be a bug in OLS itself - the root cause is probably more likely to be a bug in a WordPress plugin or a misconfiguration of OLS that causes it to run out of resources. The problem is, there's no clear guidance for isolating the issue, but it has been reported for years and must be damaging uptake of OLS.
For me, the problem is intermittent and none of the suggested configuration tweaks (for example to cpuAffinity or zlib.output_compression) have helped.
Apart from multiple "No request delivery notification has been received from LSAPI application, possible dead lock" errors, I haven't been able to find anything significant in any error logs. I don't see any hung processes when viewed with "top". Switching from memcached to redis seemed to help for a while but the problem returned. Intermittent problems are a bitch. When it happens, it seems to simultaneously affect the back end of multiple sites more that the front end - it doesn't trigger site load speed monitors and I only know about it when admins start complaining that everything runs "like treacle" or they have seen 503 timeouts. The problem can be fixed for a while (a few days) by restarting everything.
It's a production server (small VPS, 4 MB RAM, 2 cores and about 50 sites all running on Virtualmin and Ubuntu 22.04) so there is a limit to the testing that can be done. Before deploying it I did run some load tests to check resource settings, and it seemed reliable (crucially, more reliable than Apache2).
Does anyone have guidance on tweaking the numerous resource settings to avoid this? Does it affect people using the paid LiteSpeed version as well?
I accept that it may not be a bug in OLS itself - the root cause is probably more likely to be a bug in a WordPress plugin or a misconfiguration of OLS that causes it to run out of resources. The problem is, there's no clear guidance for isolating the issue, but it has been reported for years and must be damaging uptake of OLS.
For me, the problem is intermittent and none of the suggested configuration tweaks (for example to cpuAffinity or zlib.output_compression) have helped.
Apart from multiple "No request delivery notification has been received from LSAPI application, possible dead lock" errors, I haven't been able to find anything significant in any error logs. I don't see any hung processes when viewed with "top". Switching from memcached to redis seemed to help for a while but the problem returned. Intermittent problems are a bitch. When it happens, it seems to simultaneously affect the back end of multiple sites more that the front end - it doesn't trigger site load speed monitors and I only know about it when admins start complaining that everything runs "like treacle" or they have seen 503 timeouts. The problem can be fixed for a while (a few days) by restarting everything.
It's a production server (small VPS, 4 MB RAM, 2 cores and about 50 sites all running on Virtualmin and Ubuntu 22.04) so there is a limit to the testing that can be done. Before deploying it I did run some load tests to check resource settings, and it seemed reliable (crucially, more reliable than Apache2).
Does anyone have guidance on tweaking the numerous resource settings to avoid this? Does it affect people using the paid LiteSpeed version as well?
Hi,
I saw you mentioned " I don't see any hung processes when viewed with "top" ", is it possible to share the output when the 503 issue happens? For a 4 GB RAM server with 50 sites plus control panel, my guess is there are too many PHP processes running at the same time and the server does not have enough resources to handle them.
I saw you mentioned " I don't see any hung processes when viewed with "top" ", is it possible to share the output when the 503 issue happens? For a 4 GB RAM server with 50 sites plus control panel, my guess is there are too many PHP processes running at the same time and the server does not have enough resources to handle them.
Thanks for your response. I'm a bit reluctant to knowingly inflict pain on my users, but I'll do it once if we're prepared to gather all necessary information in one go. Which "resources" need to be monitored, exactly?
I should say that the "possible dead lock" messages are much more common than 503 errors, which only happen when the server gets so slow it hits timeouts.
Also, I routinely monitor CPU, RAM, network and disk usage and I don't believe the server is "overloaded" as such. Apache is quite happily serving the same sites as I speak. Before the server was commissioned I ran a whole lot of load tests using loadimpact.com and it's nowhere near the level that caused timeouts in those tests.
Having said that, it's quite clear that there's either some sort of resource issue, or a "dead lock" bug as the error message implies. Hence my question, "which resources need to be monitored?".
I'm happy to give access to the control panel if that would help.
I should say that the "possible dead lock" messages are much more common than 503 errors, which only happen when the server gets so slow it hits timeouts.
Also, I routinely monitor CPU, RAM, network and disk usage and I don't believe the server is "overloaded" as such. Apache is quite happily serving the same sites as I speak. Before the server was commissioned I ran a whole lot of load tests using loadimpact.com and it's nowhere near the level that caused timeouts in those tests.
Having said that, it's quite clear that there's either some sort of resource issue, or a "dead lock" bug as the error message implies. Hence my question, "which resources need to be monitored?".
I'm happy to give access to the control panel if that would help.