OLS not responding

ckissi

ckissi

Member
#1
#1
Hi,

Happened to me again. No response in front-end (domains on OLS) as well as OSL admin interface (ERR_TIMED_OUT)

Code: 
root@buyvm4:~# ps aux | grep lsws
root     22894  0.0  0.0   4512   836 ?        S    03:07   0:00 sh -c gdb --batch --command=/usr/local/lsws//admin/misc/gdb-bt /usr/local/lsws//bin/lshttpd /tmp/lshttpd/core* >> /tmp/m-xV0UnS
root     22895  0.0  1.9  74828 20300 ?        S    03:07   0:00 gdb --batch --command=/usr/local/lsws//admin/misc/gdb-bt /usr/local/lsws//bin/lshttpd /tmp/lshttpd/core
root     23189  0.0  0.1  16580  1956 pts/0    S+   03:25   0:00 grep --color=auto lsws
Code: 
root@buyvm4:~# service lsws status
* lsws.service - LSB: lshttpd
   Loaded: loaded (/etc/init.d/lsws; bad; vendor preset: enabled)
   Active: active (running) since Mon 2017-08-14 04:04:36 EDT; 6 days ago
     Docs: man:systemd-sysv-generator(8)
  Process: 581 ExecStart=/etc/init.d/lsws start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/lsws.service
           |-22887 lsphp
           |-22888 lsphp
           |-22894 sh -c gdb --batch --command=/usr/local/lsws//admin/misc/gdb-bt /usr/local/lsws//bin/lshttpd /tmp/lshttpd/core* >> /tmp/m-xV0UnS
           |-22895 gdb --batch --command=/usr/local/lsws//admin/misc/gdb-bt /usr/local/lsws//bin/lshttpd /tmp/lshttpd/core
           |-28143 openlitespeed (lshttpd - main
           `-28151 openlitespeed (lscgid

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
Server Error log:
Code: 
root@buyvm4:~# cat /usr/local/lsws/logs/error.log
2017-08-15 17:05:43.074 [ERROR] [198.98.55.68:59136] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
2017-08-15 20:23:53.661 [ERROR] [198.98.55.68:60884] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 97920, size: 0, next: 0
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 119376, size: 0, next: 97920
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 128640, size: 0, next: 119376
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 137856, size: 0, next: 128640
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 13248, size: 0, next: 137856
2017-08-16 03:00:08.026 [ERROR] [198.98.55.68:36192] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 154320, size: 0, next: 13248
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 38848, size: 0, next: 154320
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 117680, size: 0, next: 38848
2017-08-16 08:23:47.328 [ERROR] [198.98.55.68:39216] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
2017-08-16 14:28:25.615 [ERROR] [198.98.55.68:42256] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 150112, size: 0, next: 117680
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 152144, size: 0, next: 150112
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 157280, size: 0, next: 152144
2017-08-16 17:05:47.035 [ERROR] [198.98.55.68:43722] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
2017-08-16 20:23:56.724 [ERROR] [198.98.55.68:45072] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
2017-08-16 20:38:38.508 [ERROR] [198.98.55.68:45102] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 135168, size: 0, next: 157280
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 53184, size: 0, next: 135168
2017-08-17 08:23:55.510 [ERROR] [198.98.55.68:47482] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 61024, size: 0, next: 53184
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 115696, size: 0, next: 61024
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 108128, size: 0, next: 115696
2017-08-17 11:05:58.796 [ERROR] [198.98.55.68:48202] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 160320, size: 0, next: 108128
2017-08-17 17:01:13.469 [ERROR] [198.98.55.68:49322] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
2017-08-17 17:06:14.808 [ERROR] [198.98.55.68:49334] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 31344, size: 0, next: 160320
2017-08-17 20:24:03.786 [ERROR] [198.98.55.68:50162] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 53872, size: 0, next: 31344
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 109536, size: 0, next: 53872
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 141936, size: 0, next: 109536
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 23168, size: 0, next: 141936
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 125568, size: 0, next: 23168
2017-08-18 07:00:42.549 [ERROR] [198.98.55.68:54544] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
2017-08-18 08:23:52.613 [ERROR] [198.98.55.68:55150] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 155248, size: 0, next: 125568
2017-08-18 10:17:51.491 [ERROR] [198.98.55.68:56318] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 10880, size: 0, next: 155248
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 28672, size: 0, next: 10880
2017-08-18 17:06:25.475 [ERROR] [198.98.55.68:59924] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 8192, size: 0, next: 28672
2017-08-18 20:23:48.336 [ERROR] [198.98.55.68:33246] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
2017-08-19 06:00:09.118 [ERROR] [198.98.55.68:37932] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
2017-08-19 08:24:49.851 [ERROR] [198.98.55.68:39098] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 100320, size: 0, next: 8192
2017-08-19 17:05:55.918 [ERROR] [198.98.55.68:43298] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 19072, size: 0, next: 100320
2017-08-19 20:24:25.069 [ERROR] [198.98.55.68:44710] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
2017-08-19 22:40:03.196 [ERROR] [198.98.55.68:45340] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
2017-08-19 23:39:26.335 [ERROR] [198.98.55.68:45492] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
2017-08-20 06:01:52.591 [ERROR] [198.98.55.68:46430] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
2017-08-20 08:23:46.711 [ERROR] [198.98.55.68:46830] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
2017-08-20 17:05:49.626 [ERROR] [198.98.55.68:48386] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
[DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 66496, size: 0, next: 19072
2017-08-20 20:23:56.481 [ERROR] [198.98.55.68:49218] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
Code: 
root@buyvm4:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            479M     0  479M   0% /dev
tmpfs           100M   13M   87M  13% /run
/dev/vda1        19G  9.3G  8.5G  53% /
tmpfs           497M  184K  496M   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           497M     0  497M   0% /sys/fs/cgroup
I have to restart the lsws service to make it work again.
 
lsfoo

lsfoo

Moderator
#3
#3
Hi @ckissi

Did you get a core dump from when this occurred? It could be a SHM corruption. There should have been a core file generated based on the ps output.

Cheers,
Kevin
 
ckissi

ckissi

Member
#4
#4
Sorry Kevin, I don't get it.

Can you describe step by step what should I do when it happens again? What commands with which arguments to run?
 
lsfoo

lsfoo

Moderator
#5
#5
Please check /tmp/lshttpd/ (maybe bak-core). There is likely a core file there. If the timestamp matches about the time OLS stopped working, it was probably generated by that issue.

If this is the case, please open a private conversation with me and send me the output after the following commands:
Code: 
> gdb /usr/local/lsws/bin/openlitespeed /tmp/lshttpd/bak_core/core
> bt
 
ckissi

ckissi

Member
#7
#7
This is the list of the folder currently:

Code: 
root@buyvm4:/tmp/lshttpd# ll
drwxr-xr-x  3 nobody nogroup   4096 Aug 21 03:48 ./
drwxrwxrwt  9 root   root    208896 Aug 21 10:27 ../
-rw-r--r--  1 root   root         5 Aug 21 03:47 lshttpd.pid
srwxr-xr-x  1 nobody nogroup      0 Aug 21 03:48 lsphp56.sock=
srwxr-xr-x  1 nobody nogroup      0 Aug 21 03:48 lsphp.sock=
-rw-r--r--  1 nobody nogroup   1370 Aug 21 10:28 .rtreport
-rw-r--r--  1 root   nogroup   1198 Aug 21 03:47 .status
drwx------ 18 nobody nogroup   4096 Aug 21 03:48 swap/
No dump there. So probably I have to wait when it'll happen again.
 
lsfoo

lsfoo

Moderator
#8
#8
Just to confirm, you have gdb installed there correct?

In that case, yeah - it looks like it's trying to create a core file, but did not yet.
 
ckissi

ckissi

Member
#9
#9
Code: 
root@buyvm4:~# gdb --version
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
 
lsfoo

lsfoo

Moderator
#10
#10
Well, until it creates a core file, unfortunately, we cannot do much to figure out what the issue is.

Please try deleting the contents of /dev/shm/ols in the meantime and restart the server, hopefully the issue will present itself.
 
ckissi

ckissi

Member
#11
#11
What these files actually are ?
Code: 
-rw------- 1 nobody nogroup   8192 Aug 21 03:38 adns_cache.lock
-rw------- 1 nobody nogroup 131072 Aug 21 11:35 adns_cache.shm
 
You must log in or register to reply here.
Top