So one of our servers running librenms lost power and since regaining power back we have been seeing issues with librenms and mysql database. Here is the output from validate.php:
./validate.php
| Component | Version |
|---|---|
| LibreNMS | 23.11.0-64-g2ed5afc82 (2023-12-31T12:19:32+00:00) |
| DB Schema | Not Connected (0) |
| PHP | 8.1.10 |
| Python | 3.8.10 |
| Database | Not Connected |
| RRDTool | 1.7.2 |
| SNMP | 5.8 |
| =========================================== |
[OK] Composer Version: 2.6.6
[OK] Dependencies up-to-date.
Could not connect to database, check logs/librenms.log.
Logs from librenms.log:
Next Illuminate\Database\QueryException: SQLSTATE[HY000] [2002] Connection refused (Connection: mysql, SQL: select * from config where (config_name = snmp.unescape) limit 1) in /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Connection.php:795
Stack trace:
If we look at the mysql server running on the same host we see its stuck at “activating”
service mysqld status
â—Ź mariadb.service - MariaDB 10.3.38 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: signal) since Fri 2024-01-05 10:06:10 UTC; 4s ago
Docs: man:mysqld(8)
systemd - MariaDB Knowledge Base
Process: 35623 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 35624 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 35626 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=cd /usr/bin/..; /usr/bin/galera_recovery; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited>
Process: 35679 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=killed, signal=SEGV)
Main PID: 35679 (code=killed, signal=SEGV)
Status: “Starting final batch to recover 627 pages from redo log”
Let me know if there is any other output I can provide to help with the troubleshooting / solution here.