SQLSTATE[HY000] [2002] Connection refused after 18 hours

Hi Team,

Fresh librenms container on Ubuntu 22.04 under Proxmox hypervisor. Imported over rrd files and database from previous Centos 7 install. After around 18 hours it keeps falling over.

in /etc/mysql/my.cnf I’ve increased max_connections but this hasn’t helped.

[mysqld]
max_connections = 1000
librenms@librenms:~$ ./validate.php
===========================================
Component | Version
--------- | -------
LibreNMS  | 23.2.0-44-g0a889ad36 (2023-03-24T14:29:53+13:00)
DB Schema | Not Connected (0)
PHP       | 8.1.2
Python    | 3.10.4
Database  | Not Connected
RRDTool   | 1.7.2
SNMP      | 5.9.1
===========================================

[OK]    Composer Version: 2.5.4
[OK]    Dependencies up-to-date.
Could not connect to database, check logs/librenms.log.

ps -axjf
https://p.libren.ms/view/8efefb38

librenms.log seems to stop polling around 02:30.
https://p.libren.ms/view/7ac90268

have to check my mysql exploded? mysql.err file or logs? etc

syslog

Mar 27 04:34:01 librenms CRON[1599143]: (librenms) CMD (   cd /opt/librenms/ && php artisan schedule:run >> /dev/null 2>&1)
Mar 27 04:34:01 librenms CRON[1599142]: (librenms) CMD (   /opt/librenms/alerts.php >> /dev/null 2>&1)
Mar 27 04:35:01 librenms CRON[1599181]: (librenms) CMD (   /opt/librenms/check-services.php >> /dev/null 2>&1)
Mar 27 04:35:01 librenms CRON[1599184]: (librenms) CMD (   /opt/librenms/discovery.php -h new >> /dev/null 2>&1)
Mar 27 04:35:01 librenms CRON[1599182]: (librenms) CMD (   /opt/librenms/cronic /opt/librenms/poller-wrapper.py 16)
Mar 27 04:35:01 librenms CRON[1599183]: (librenms) CMD (   /opt/librenms/alerts.php >> /dev/null 2>&1)
Mar 27 04:35:01 librenms CRON[1599180]: (librenms) CMD (   cd /opt/librenms/ && php artisan schedule:run >> /dev/null 2>&1)
Mar 27 04:35:01 librenms CRON[1599185]: (librenms) CMD (   /opt/librenms/poll-billing.php >> /dev/null 2>&1)
Mar 27 04:35:02 librenms mariadbd[177]: 2023-03-27  4:35:02 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Mar 27 04:35:02 librenms mariadbd[177]: message repeated 115 times: [ 2023-03-27  4:35:02 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)]
Mar 27 04:35:03 librenms mariadbd[177]: 2023-03-27  4:35:03 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Mar 27 04:35:03 librenms mariadbd[177]: message repeated 92 times: [ 2023-03-27  4:35:03 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)]
Mar 27 04:35:04 librenms mariadbd[177]: 2023-03-27  4:35:04 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Mar 27 04:35:04 librenms mariadbd[177]: message repeated 372 times: [ 2023-03-27  4:35:04 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)]
Mar 27 04:35:05 librenms mariadbd[177]: 2023-03-27  4:35:05 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Mar 27 04:35:05 librenms mariadbd[177]: message repeated 147 times: [ 2023-03-27  4:35:05 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)]

etc

MariaDB [(none)]>  SHOW VARIABLES LIKE 'max_connections';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 2048  |
+-----------------+-------+
1 row in set (0.001 sec)

MariaDB [(none)]>  SHOW STATUS WHERE `variable_name` = 'Threads_connected';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| Threads_connected | 165   |
+-------------------+-------+
1 row in set (0.001 sec)

You can also setup some snmpd and script so you get some basic stats in

Also since new install might be worth running

So far so good. I’ve got SNMP monitoring. rrdcached has massively reduced diskIO. I had ran MySQLTuner on previous Centos hosted LibreNMS but I just want it stable first before tweaking.

root@librenms:~# uptime
 08:23:59 up 23:01,  1 user,  load average: 0.36, 0.91, 0.89
MariaDB [(none)]> show variables like 'max_connections';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 1000  |
+-----------------+-------+
1 row in set (0.000 sec)

MariaDB [(none)]> show status where 'variable_name' = 'Threads_connected';
Empty set (0.000 sec)

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.