Hi Team,
Fresh librenms container on Ubuntu 22.04 under Proxmox hypervisor. Imported over rrd files and database from previous Centos 7 install. After around 18 hours it keeps falling over.
in /etc/mysql/my.cnf I’ve increased max_connections but this hasn’t helped.
[mysqld]
max_connections = 1000
librenms@librenms:~$ ./validate.php
===========================================
Component | Version
--------- | -------
LibreNMS | 23.2.0-44-g0a889ad36 (2023-03-24T14:29:53+13:00)
DB Schema | Not Connected (0)
PHP | 8.1.2
Python | 3.10.4
Database | Not Connected
RRDTool | 1.7.2
SNMP | 5.9.1
===========================================
[OK] Composer Version: 2.5.4
[OK] Dependencies up-to-date.
Could not connect to database, check logs/librenms.log.
ps -axjf
https://p.libren.ms/view/8efefb38
librenms.log seems to stop polling around 02:30.
https://p.libren.ms/view/7ac90268
have to check my mysql exploded? mysql.err file or logs? etc
syslog
Mar 27 04:34:01 librenms CRON[1599143]: (librenms) CMD ( cd /opt/librenms/ && php artisan schedule:run >> /dev/null 2>&1)
Mar 27 04:34:01 librenms CRON[1599142]: (librenms) CMD ( /opt/librenms/alerts.php >> /dev/null 2>&1)
Mar 27 04:35:01 librenms CRON[1599181]: (librenms) CMD ( /opt/librenms/check-services.php >> /dev/null 2>&1)
Mar 27 04:35:01 librenms CRON[1599184]: (librenms) CMD ( /opt/librenms/discovery.php -h new >> /dev/null 2>&1)
Mar 27 04:35:01 librenms CRON[1599182]: (librenms) CMD ( /opt/librenms/cronic /opt/librenms/poller-wrapper.py 16)
Mar 27 04:35:01 librenms CRON[1599183]: (librenms) CMD ( /opt/librenms/alerts.php >> /dev/null 2>&1)
Mar 27 04:35:01 librenms CRON[1599180]: (librenms) CMD ( cd /opt/librenms/ && php artisan schedule:run >> /dev/null 2>&1)
Mar 27 04:35:01 librenms CRON[1599185]: (librenms) CMD ( /opt/librenms/poll-billing.php >> /dev/null 2>&1)
Mar 27 04:35:02 librenms mariadbd[177]: 2023-03-27 4:35:02 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Mar 27 04:35:02 librenms mariadbd[177]: message repeated 115 times: [ 2023-03-27 4:35:02 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)]
Mar 27 04:35:03 librenms mariadbd[177]: 2023-03-27 4:35:03 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Mar 27 04:35:03 librenms mariadbd[177]: message repeated 92 times: [ 2023-03-27 4:35:03 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)]
Mar 27 04:35:04 librenms mariadbd[177]: 2023-03-27 4:35:04 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Mar 27 04:35:04 librenms mariadbd[177]: message repeated 372 times: [ 2023-03-27 4:35:04 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)]
Mar 27 04:35:05 librenms mariadbd[177]: 2023-03-27 4:35:05 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Mar 27 04:35:05 librenms mariadbd[177]: message repeated 147 times: [ 2023-03-27 4:35:05 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)]
etc
MariaDB [(none)]> SHOW VARIABLES LIKE 'max_connections';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 2048 |
+-----------------+-------+
1 row in set (0.001 sec)
MariaDB [(none)]> SHOW STATUS WHERE `variable_name` = 'Threads_connected';
+-------------------+-------+
| Variable_name | Value |
+-------------------+-------+
| Threads_connected | 165 |
+-------------------+-------+
1 row in set (0.001 sec)
You can also setup some snmpd and script so you get some basic stats in
Also since new install might be worth running
So far so good. I’ve got SNMP monitoring. rrdcached has massively reduced diskIO. I had ran MySQLTuner on previous Centos hosted LibreNMS but I just want it stable first before tweaking.
root@librenms:~# uptime
08:23:59 up 23:01, 1 user, load average: 0.36, 0.91, 0.89
MariaDB [(none)]> show variables like 'max_connections';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 1000 |
+-----------------+-------+
1 row in set (0.000 sec)
MariaDB [(none)]> show status where 'variable_name' = 'Threads_connected';
Empty set (0.000 sec)