Ahsan
19 November 2024 14:53
1
Hi
.sudo -u librenms ./validate.php
===========================================
Component | Version
--------- | -------
LibreNMS | 24.10.1-50-g5f59ae051 (2024-11-17T20:22:32+05:00)
DB Schema | Not Connected (0)
PHP | 8.3.12
Python | 3.10.12
Database | Not Connected
RRDTool | 1.7.2
SNMP | 5.9.1
===========================================
[OK] Composer Version: 2.8.2
[OK] Dependencies up-to-date.
[FAIL] Error connecting to your database.
[FAIL] Error connecting to your database.
[FAIL] Locking server issue: SQLSTATE[HY000] [2002] Connection refused (Connection: mysql, SQL: update `cache_locks` set `owner` = o2AGTjGxeGRVkQu8, `expiration` = 1732027797 where `key` = laravel_cache_dist_test_validation and (`owner` = o2AGTjGxeGRVkQu8 or `expiration` <= 1732027792))
[OK] Redis is unavailable
[OK] rrd_dir is writable
[OK] rrdtool version ok
In Connection.php line 829:
SQLSTATE[HY000] [2002] Connection refused (Connection: mysql, SQL: select * from `cache` where `key` = laravel_cache_scheduler_working limit 1)
In Connector.php line 65:
SQLSTATE[HY000] [2002] Connection refused
df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 392M 1.6M 390M 1% /run
/dev/sda2 118G 26G 87G 23% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
efivarfs 256K 58K 194K 23% /sys/firmware/efi/efivars
/dev/sda1 511M 6.1M 505M 2% /boot/efi
tmpfs 392M 80K 392M 1% /run/user/128
tmpfs 392M 72K 392M 1% /run/user/1000
Ahsan
19 November 2024 15:02
2
though i resolved the issue by restarting the service using below command
but why it happened ? any thing that i need to ensure to avoid its occurrence ?
laf
19 November 2024 16:00
3
This isn’t a LibreNMS issue really and rather a standard Linux sysadmin thing. Check all available logs and next time check that the service is actually already running.
Hi @laf , I am facing the same issue, checking the status of mysql indicates its starting and stopping every few seconds, Could this be a bug with the latest commit as this is when the issue started happening.
root@librenms:/opt/librenms# git log
commit 354aa370d1e9a1601ea0dd05f6c4edf629a999aa (HEAD -> master, origin/master, origin/HEAD)
Author: Kevin Zink <[email protected] >
Date: Wed Nov 20 01:05:02 2024 +0100
Hirschmann: Convert state sensors from inc.php to yaml (#16704)
* Hirschmann: Convert sensors from inc.php to yaml
* fix typo
* Fix index to not break historical data
* Update hirschmann.json
* Update hirschmann_ms4128.json
---------
Co-authored-by: Neil Lathwood <[email protected] >
Error log
root@librenms:~# tail -F /opt/librenms/logs/librenms.log
#28 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(1016): array_walk()
#29 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/BootProviders.php(17): Illuminate\Foundation\Application->boot()
#30 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(263): Illuminate\Foundation\Bootstrap\BootProviders->bootstrap()
#31 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(447): Illuminate\Foundation\Application->bootstrapWith()
#32 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(199): Illuminate\Foundation\Console\Kernel->bootstrap()
#33 /opt/librenms/app/Console/Kernel.php(66): Illuminate\Foundation\Console\Kernel->handle()
#34 /opt/librenms/artisan(35): App\Console\Kernel->handle()
#35 {main}
SQLSTATE[HY000] [2002] Connection refused (Connection: mysql, SQL: delete from `cache` where `key` = laravel_cache_illuminate:schedule:interrupt) {"exception":"[object] (Illuminate\\Database\\QueryException(code: 2002): SQLSTATE[HY000] [2002] Connection refused (Connection: mysql, SQL: delete from `cache` where `key` = laravel_cache_illuminate:schedule:interrupt) at /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Connection.php:829)
[previous exception] [object] (PDOException(code: 2002): SQLSTATE[HY000] [2002] Connection refused at /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php:65)"}
tack trace:
#0 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Connection.php(983): Illuminate\Database\Connection->runQueryCallback()
#1 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Connection.php(962): Illuminate\Database\Connection->tryAgainIfCausedByLostConnection()
#2 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Connection.php(785): Illuminate\Database\Connection->handleQueryException()
#3 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Connection.php(414): Illuminate\Database\Connection->run()
#4 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2913): Illuminate\Database\Connection->select()
#5 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2902): Illuminate\Database\Query\Builder->runSelect()
#6 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(3456): Illuminate\Database\Query\Builder->Illuminate\Database\Query\{closure}()
#7 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2901): Illuminate\Database\Query\Builder->onceWithColumns()
#8 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php(739): Illuminate\Database\Query\Builder->get()
#9 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php(723): Illuminate\Database\Eloquent\Builder->getModels()
#10 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Concerns/BuildsQueries.php(333): Illuminate\Database\Eloquent\Builder->get()
#11 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php(568): Illuminate\Database\Eloquent\Builder->first()
#12 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php(600): Illuminate\Database\Eloquent\Builder->firstOrCreate()
#13 /opt/librenms/vendor/laravel/framework/src/Illuminate/Support/Traits/ForwardsCalls.php(23): Illuminate\Database\Eloquent\Builder->updateOrCreate()
#14 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php(2335): Illuminate\Database\Eloquent\Model->forwardCallTo()
#15 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php(2347): Illuminate\Database\Eloquent\Model->__call()
#16 /opt/librenms/LibreNMS/Config.php(260): Illuminate\Database\Eloquent\Model::__callStatic()
#17 /opt/librenms/LibreNMS/Config.php(460): LibreNMS\Config::persist()
#18 /opt/librenms/LibreNMS/Config.php(61): LibreNMS\Config::processConfig()
#19 /opt/librenms/app/Providers/ConfigServiceProvider.php(27): LibreNMS\Config::load()
#20 /opt/librenms/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): App\Providers\ConfigServiceProvider->boot()
#21 /opt/librenms/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#22 /opt/librenms/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure()
#23 /opt/librenms/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod()
#24 /opt/librenms/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\Container\BoundMethod::call()
#25 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(1036): Illuminate\Container\Container->call()
#26 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(1017): Illuminate\Foundation\Application->bootProvider()
#27 [internal function]: Illuminate\Foundation\Application->Illuminate\Foundation\{closure}()
#28 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(1016): array_walk()
#29 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/BootProviders.php(17): Illuminate\Foundation\Application->boot()
#30 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(263): Illuminate\Foundation\Bootstrap\BootProviders->bootstrap()
#31 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(447): Illuminate\Foundation\Application->bootstrapWith()
#32 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(199): Illuminate\Foundation\Console\Kernel->bootstrap()
#33 /opt/librenms/app/Console/Kernel.php(66): Illuminate\Foundation\Console\Kernel->handle()
#34 /opt/librenms/artisan(35): App\Console\Kernel->handle()
#35 {main}
MySQL/MariaDB restarting every few seconds
root@librenms:~# systemctl status mysql
● mariadb.service - MariaDB 10.4.31 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: activating (auto-restart) (Result: signal) since Wed 2024-11-20 05:59:19 CAT; 4s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 1580959 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 1580961 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 1580963 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_ST>
Process: 1581052 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=killed, signal=ABRT)
Process: 1581087 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 1581089 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
Main PID: 1581052 (code=killed, signal=ABRT)
Status: "Taking your SQL requests now..."
Nov 20 05:59:19 librenms.liquidtelecom.zm systemd[1]: mariadb.service: Failed with result 'signal'.
root@librenms:~# systemctl status mysql
● mariadb.service - MariaDB 10.4.31 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: active (running) since Wed 2024-11-20 05:59:25 CAT; 1s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 1581177 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 1581185 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 1581188 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_ST>
Process: 1581313 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 1581315 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
Main PID: 1581278 (mysqld)
Status: "Taking your SQL requests now..."
Tasks: 32 (limit: 77026)
Memory: 113.8M
CGroup: /system.slice/mariadb.service
├─1581278 /usr/sbin/mysqld
└─1581331 addr2line -C -f -e /lib/x86_64-linux-gnu/libpthread.so.0
root@librenms:~# systemctl status mysql
● mariadb.service - MariaDB 10.4.31 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: activating (auto-restart) (Result: signal) since Wed 2024-11-20 05:59:30 CAT; 717ms ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 1581177 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 1581185 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 1581188 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_ST>
Process: 1581278 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=killed, signal=ABRT)
Process: 1581313 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 1581315 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
Main PID: 1581278 (code=killed, signal=ABRT)
Status: "Taking your SQL requests now..."
root@librenms:~#
laf
20 November 2024 12:37
5
No, that wouldn’t trigger mysql to restart, LibreNMS in general is unlikely to do that. You need to check the mysql/mariadb log files.
Ahsan
21 November 2024 17:35
6
this is happneing frequently. i have to face this problem on daily basis. and have to manually restart mariadb. any idea what could be the root cause for this.
MariaDB is crashing, unfortunately, we don’t know how to help you with MariaDB, only LibreNMS.
You’ll need to check the MariaDB logs and search the errors or ask in the MariaDB community for help.
Ahsan
25 November 2024 14:27
8
نومبر 19 20:12:42 MyNMS-virtual-machine mariadbd[1719708]: 2024-11-19 20:12:42 638 [Warning] Access denied for user ‘soaknms’@‘localhost’ (using password: NO)
laf
25 November 2024 15:49
9
Ahsan:
oom-kill
That’s probably your issue, have a google for it and I’m sure you’ll find lots on what to do to resolve it.
jaysen
26 November 2024 04:26
10
I have encountered both of these issues in past installs and I see two issues going on.
First, access is being denied by MariaDB to the user librenms@localhost. To fix this, just add another user called librenms@localhost and make sure you grant all privileges to the librenms database to this new user then flush privileges. The MySQL commands to create the user and privileges are in the setup guide. Make sure you use the same password that you used when you did your installation. This should solve your access denied issue.
Second, the oom-killer is due to your server running out of memory. If you are running your entire installation on a small VPS. You may need to upgrade it to the next server tier to include more RAM. You can use the top and free commands to inspect your processes and memory usage. If it shows you are using swap memory then I highly recommend you consider upgrading. I’d start with the simplest thing by fixing the access denied issue first.
There are of course many other MySQL tweaks you can do to tune the server but I’ve always found that to be a huge pain. Start with the simple stuff and see what happens. As always, make a backup before making any changes.
