Error establishing a database connection

[mello555]

Hi,

I have been happily running the OLS Wordpress VM on the GCP platform. Suddenly I started getting the "Error establishing a database connection" error.

I tried including the define('WP_ALLOW_REPAIR', true); in WP-config.php file and accessing and fix through mysite.com/wp-admin/maint/repair.php but I get same error.

Then I tried accessing PHPMyAdmin and login to get a manual backup. Unfortunately, the WordPress USERNAME and PASSWORD in wp-config.php file doesn't seem to work.

In SSH I tried to check the MariaDB status and I get following error message

mariadb.service - MariaDB 10.3.16 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: failed (Result: exit-code) since Tue 2019-12-10 05:47:08 UTC; 7h ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 4428 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, st
  Process: 4350 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recover
  Process: 4348 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SU
  Process: 4347 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUC
 Main PID: 4428 (code=exited, status=1/FAILURE)
   Status: "MariaDB server is down"
    Tasks: 3 (limit: 661)
   CGroup: /system.slice/mariadb.service
           └─32608 /usr/sbin/mysqld
  Process: 4347 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
 Main PID: 4428 (code=exited, status=1/FAILURE)
   Status: "MariaDB server is down"
    Tasks: 3 (limit: 661)
   CGroup: /system.slice/mariadb.service
           └─32608 /usr/sbin/mysqld

Dec 10 05:43:34 openlitespeed-wordpress-2-vm systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Dec 10 05:43:34 openlitespeed-wordpress-2-vm systemd[1]: Starting MariaDB 10.3.16 database server...
Dec 10 05:43:35 openlitespeed-wordpress-2-vm systemd[1]: mariadb.service: Found left-over process 32608 (mysqld) in control group while starting unit. Ignoring.
Dec 10 05:43:35 openlitespeed-wordpress-2-vm systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Dec 10 05:43:35 openlitespeed-wordpress-2-vm mysqld[4428]: 2019-12-10  5:43:35 0 [Note] /usr/sbin/mysqld (mysqld 10.3.16-MariaDB-1:10.3.16+maria~bionic) starting as process 4428 ...
Dec 10 05:44:07 openlitespeed-wordpress-2-vm systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Dec 10 05:45:38 openlitespeed-wordpress-2-vm systemd[1]: mariadb.service: State 'stop-sigterm' timed out. Skipping SIGKILL.
Dec 10 05:47:08 openlitespeed-wordpress-2-vm systemd[1]: mariadb.service: State 'stop-final-sigterm' timed out. Skipping SIGKILL. Entering failed mode.
Dec 10 05:47:08 openlitespeed-wordpress-2-vm systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 10 05:47:08 openlitespeed-wordpress-2-vm systemd[1]: Failed to start MariaDB 10.3.16 database server.

I tried restarting the server but I'm unable to do so.

I appreciate if anyone can help.

 

[mello555]

HI,

I fixed the problem by restarting the whole server through system reboot.

Lessons learnt:

1. When we get Error establishing a database connection - ensure database server is running.
2. If Database server is not running and if there are errors with the MySQL/MariaDB, try rebooting the system

   $sudo reboot

   for ubuntu in my case

 

[David]

Thank you for your port.

 

[Cold-Egg]

Hi @mello555 ,

Also check syslog maybe helps if it happens again.

Best

 

[lskagan]

In this specific case it is possible MariaDB was killed due to OOM issues.

You can try `sudo systemctl restart mariadb` if that doesnt work you can run `sudo ps -ef | grep 'mysql'` get the PID then run `kill -9 PID` and then `sudo systemctl start mariadb` which should get mariadb back up without having to reboot the whole server.

 

[woocommerce_dev]

Facing similar issue when querying woocomerce admin report section. Can someone help me how to root cause and have a permanent fix? Also can there be a cron to auto restart the service?

@Cold-Egg / @lskagan pl. suggest.

 

[Cold-Egg]

You can try to use the monit tool to monitor and auto-restart the SQL service.

 

[woocommerce_dev]

Sure can you pl. share any guide, I found this one - https://www.srapsware.com/linux/how-to/2021/08/21/openlitespeed-configure-with-monit-to-monitor/

BTW sharing logs:

Apr 6 05:12:58 cricket systemd[1]: mariadb.service: A process of this unit has been killed by the OOM killer.
Apr 6 05:12:58 cricket systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
Apr 6 05:12:58 cricket systemd[1]: mariadb.service: Failed with result 'oom-kill'.
Apr 6 05:12:58 cricket systemd[1]: mariadb.service: Consumed 1h 57min 38.980s CPU time.
Apr 6 05:12:58 cricket kernel: [1623241.035257] [ 917] 112 917 279484 9082 942080 75469 0 mariadbd
Apr 6 05:12:58 cricket kernel: [1623241.035460] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=lshttpd.service,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mariadbd,pid=917,uid=112
Apr 6 05:12:58 cricket kernel: [1623241.035559] Out of memory: Killed process 917 (mariadbd) total-vm:1117936kB, anon-rss:36328kB, file-rss:0kB, shmem-rss:0kB, UID:112 pgtables:920kB oom_score_adj:0

 

[woocommerce_dev]

Curious where is the error log being stored by default for one click install of DigitalOcean? I don't see a single file in /var/log/mysql. Also I was thinking to understand where to tweak innodb_buffer_pool_size in OLS? For now, I have made changes to /etc/mysql/mariadb.conf.d/50-server.cnf and disabled the comments and set following values.

log_error = /var/log/mysql/error.log
innodb_buffer_pool_size = 750M

 

[Cold-Egg]

Out of memory: Killed process 917 (mariadbd) means memory is not enough and the process got killed.

Your setup should be fine, by adding the log_error to the conf file.

 

[Avidixx]

After extensively searching online and frustrating talking with ChatGPT-4, I decided to try updating the server as a potential solution. I executed the following commands:

sudo apt-get update

sudo apt-get upgrade -y

To my relief, this resolved the issue. I'm documenting this solution here for future reference. Should I encounter a similar problem again, I hope to find this post through a quick Google search and remind myself of the fix.

 

[mister_x]

Can you please share your system's resources?

Doing apt update && apt upgrade is not a problem-fixing command.
It is for updating and it is a norm to run this command frequently for updating and security purposes.