Librenms - Could not connect to database Error

So one of our servers running librenms lost power and since regaining power back we have been seeing issues with librenms and mysql database. Here is the output from validate.php:

./validate.php

Component Version
LibreNMS 23.11.0-64-g2ed5afc82 (2023-12-31T12:19:32+00:00)
DB Schema Not Connected (0)
PHP 8.1.10
Python 3.8.10
Database Not Connected
RRDTool 1.7.2
SNMP 5.8
===========================================

[OK] Composer Version: 2.6.6
[OK] Dependencies up-to-date.
Could not connect to database, check logs/librenms.log.

Logs from librenms.log:
Next Illuminate\Database\QueryException: SQLSTATE[HY000] [2002] Connection refused (Connection: mysql, SQL: select * from config where (config_name = snmp.unescape) limit 1) in /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Connection.php:795
Stack trace:

If we look at the mysql server running on the same host we see its stuck at “activating”

service mysqld status
● mariadb.service - MariaDB 10.3.38 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: signal) since Fri 2024-01-05 10:06:10 UTC; 4s ago
Docs: man:mysqld(8)
systemd - MariaDB Knowledge Base
Process: 35623 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 35624 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 35626 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=cd /usr/bin/..; /usr/bin/galera_recovery; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited>
Process: 35679 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=killed, signal=SEGV)
Main PID: 35679 (code=killed, signal=SEGV)
Status: “Starting final batch to recover 627 pages from redo log”

Let me know if there is any other output I can provide to help with the troubleshooting / solution here.

Looks like mysql isn’t starting and needs to be recovered. You could try: InnoDB Recovery Modes - MariaDB Knowledge Base

1 Like