Updating when using the dispatcher service

systeembeheerder · 17 June 2024 11:02

I have updates configured:

librenms@nms:~$ lnms config:get  --dump | jq '{update, update_channel}'
{
  "update": true,
  "update_channel": "release"
}

The service is running (for more than a month now):

librenms@nms:~$ systemctl status librenms.service
● librenms.service - LibreNMS SNMP Poller Service
     Loaded: loaded (/etc/systemd/system/librenms.service; enabled; preset: enabled)
     Active: active (running) since Wed 2024-05-08 06:10:53 CEST; 1 month 9 days ago
   Main PID: 492 (python3)
      Tasks: 53 (limit: 2158)
     Memory: 145.6M
        CPU: 6d 18h 56min 43.178s
     CGroup: /system.slice/librenms.service
             └─492 /usr/bin/python3 /opt/librenms/librenms-service.py -v

yet i am still not up-to-date. 24.5 and 24.6 have been released in the meantime.

librenms@nms:~$ ./validate.php
===========================================
Component | Version
--------- | -------
LibreNMS  | 24.4.1 (2024-04-20T16:26:51+02:00)
DB Schema | 2024_04_29_183605_custom_maps_drop_background_suffix_and_background_version (294)
PHP       | 8.2.18
Python    | 3.11.2
Database  | MariaDB 10.11.6-MariaDB-0+deb12u1
RRDTool   | 1.7.2
SNMP      | 5.9.3
===========================================

[OK]    Composer Version: 2.7.7
[OK]    Dependencies up-to-date.
[OK]    Database connection successful

I was assured, by @murrant, the service should do all the maintenance and update tasks

Who can help me out?

murrant · 21 June 2024 05:00

Make sure you didn’t disable maintenance in the poller settings:

systeembeheerder · 21 June 2024 07:01

Daily maintenance is enabled.

Looks like librenms.service has some trouble using mysql:

adm@nms:~$ sudo journalctl -u librenms  | grep -v Poller | grep pymysql.err | tail -n 10
Jun 21 07:15:00 nms librenms-service.py[492]: pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
Jun 21 07:25:00 nms librenms-service.py[492]: pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
Jun 21 07:35:00 nms librenms-service.py[492]: pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
Jun 21 07:45:00 nms librenms-service.py[492]: pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
Jun 21 07:55:00 nms librenms-service.py[492]: pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
Jun 21 08:05:00 nms librenms-service.py[492]: pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
Jun 21 08:15:00 nms librenms-service.py[492]: pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
Jun 21 08:25:00 nms librenms-service.py[492]: pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
Jun 21 08:35:00 nms librenms-service.py[492]: pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
Jun 21 08:45:00 nms librenms-service.py[492]: pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")

note the exact 10 minute interval. The odd thing is, polling and discovery seem to run just fine.

Jun 21 08:43:30 nms librenms-service.py[492]: Poller_17-17(INFO):Polling device 388
Jun 21 08:43:33 nms librenms-service.py[492]: Poller_17-23(INFO):Completed poller run for 1036 (17) in 3.53s
Jun 21 08:43:34 nms librenms-service.py[492]: Poller_17-17(INFO):Completed poller run for 388 (17) in 3.77s
Jun 21 08:43:42 nms librenms-service.py[492]: Poller_17-23(INFO):Polling device 647
Jun 21 08:43:45 nms librenms-service.py[492]: Poller_17-23(INFO):Completed poller run for 647 (17) in 3.43s
Jun 21 08:44:01 nms librenms-service.py[492]: Poller_17-24(INFO):Polling device 1477
Jun 21 08:44:16 nms librenms-service.py[492]: Poller_17-20(INFO):Polling device 1035
Jun 21 08:44:16 nms librenms-service.py[492]: Poller_17-24(INFO):Completed poller run for 1477 (17) in 14.45s
Jun 21 08:44:20 nms librenms-service.py[492]: Poller_17-20(INFO):Completed poller run for 1035 (17) in 3.97s
Jun 21 08:45:33 nms librenms-service.py[492]: Poller_17-6(INFO):Polling device 1487

all dependencies seem to be installed correctly

librenms@nms:~$ pip3  install  -r /opt/librenms/requirements.txt  --user --break-system-packages
Requirement already satisfied: PyMySQL!=1.0.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 1)) (1.0.2)
Requirement already satisfied: python-dotenv in /usr/lib/python3/dist-packages (from -r requirements.txt (line 2)) (0.21.0)
Requirement already satisfied: redis>=4.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 3)) (4.3.4)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from -r requirements.txt (line 4)) (66.1.1)
Requirement already satisfied: psutil>=5.6.0 in ./.local/lib/python3.11/site-packages (from -r requirements.txt (line 5)) (5.9.5)
Requirement already satisfied: command_runner>=1.3.0 in ./.local/lib/python3.11/site-packages (from -r requirements.txt (line 6)) (1.5.0)

full error

Jun 21 09:04:57 nms librenms-service.py[492]: Alerting(INFO):Completed alerting run for alerts in 0.36s
Jun 21 09:05:00 nms librenms-service.py[492]: performance(INFO):Counting up time spent polling
Jun 21 09:05:00 nms librenms-service.py[492]: performance(CRITICAL):DB Connection exception (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
Jun 21 09:05:00 nms librenms-service.py[492]: performance(CRITICAL):Unable to log performance statistics - is the database still online?
Jun 21 09:05:00 nms librenms-service.py[492]: Traceback (most recent call last):
Jun 21 09:05:00 nms librenms-service.py[492]:   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 756, in _write_bytes
Jun 21 09:05:00 nms librenms-service.py[492]:     self._sock.sendall(data)
Jun 21 09:05:00 nms librenms-service.py[492]: ConnectionResetError: [Errno 104] Connection reset by peer
Jun 21 09:05:00 nms librenms-service.py[492]: During handling of the above exception, another exception occurred:
Jun 21 09:05:00 nms librenms-service.py[492]: Traceback (most recent call last):
Jun 21 09:05:00 nms librenms-service.py[492]:   File "/opt/librenms/LibreNMS/service.py", line 848, in log_performance_stats
Jun 21 09:05:00 nms librenms-service.py[492]:     self._db.query(
Jun 21 09:05:00 nms librenms-service.py[492]:   File "/opt/librenms/LibreNMS/__init__.py", line 299, in query
Jun 21 09:05:00 nms librenms-service.py[492]:     cursor.execute(query, args)
Jun 21 09:05:00 nms librenms-service.py[492]:   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 148, in execute
Jun 21 09:05:00 nms librenms-service.py[492]:     result = self._query(query)
Jun 21 09:05:00 nms librenms-service.py[492]:              ^^^^^^^^^^^^^^^^^^
Jun 21 09:05:00 nms librenms-service.py[492]:   File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 310, in _query
Jun 21 09:05:00 nms librenms-service.py[492]:     conn.query(q)
Jun 21 09:05:00 nms librenms-service.py[492]:   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 547, in query
Jun 21 09:05:00 nms librenms-service.py[492]:     self._execute_command(COMMAND.COM_QUERY, sql)
Jun 21 09:05:00 nms librenms-service.py[492]:   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 814, in _execute_command
Jun 21 09:05:00 nms librenms-service.py[492]:     self._write_bytes(packet)
Jun 21 09:05:00 nms librenms-service.py[492]:   File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 759, in _write_bytes
Jun 21 09:05:00 nms librenms-service.py[492]:     raise err.OperationalError(
Jun 21 09:05:00 nms librenms-service.py[492]: pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")

systeembeheerder · 25 June 2024 07:49

solved.

All python dependencies were installed as user: pip3 install --user -r requirements.txt as suggested by scripts/composer_wrapper.php.

Removing all those, installing PyMySQL, pythin-dotenv, redis, setuptools and psutil from Debian packages, installing command_runner as root with --break-system-packages, everything works as intended.

system · 2 July 2024 07:49

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.