Rrd cached issue

Tags: #<Tag:0x00007f707dfe98f0>

We have two servers librenms and librepoller01. librenms has MySQL, Memcached and RRDcacheD and the poller is configured to use the primary server for memcached, mysql and rrdcached. This setup was working, but then we had our systems team perform server upgrades on librepoller01 and this created an issue with rrdcached. Now none of the memory graphs work and we get errors in rrdcached. We had the systems team roll back the snapshot, but the error didn’t go away. Now when we run a poller.php on a host on the librepoller01 system we see “ERROR: rrdcached: Unknown command: last” and the logs have lots of lines like “Feb 16 14:31:29 librenms rrdcached[7186]: handle_request_update: stat (/opt/librenms/rrd/setzer293/storage-hrstorage-_dev_md19,_mounted_on___tmp.rrd) failed.”

[[email protected] librenms]# sudo -u librenms ./validate.php

Component Version
LibreNMS 21.2.0-1-g01019cfc0
DB Schema 2021_02_08_224355_fix_invalid_dates (199)
PHP 7.4.12
Python 3.6.8
MySQL 5.5.60-MariaDB
RRDTool 1.7.0
SNMP NET-SNMP 5.8

====================================

[OK] Composer Version: 2.0.9
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[INFO] Detected Python Wrapper
[OK] Connection to memcached is ok

[[email protected] librenms]# sudo -u librenms ./validate.php

Component Version
LibreNMS 21.2.0-1-g01019cf
DB Schema 2021_02_08_224355_fix_invalid_dates (199)
PHP 7.3.12
Python 3.6.8
MySQL 5.5.60-MariaDB
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 2.0.9
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[INFO] Detected Python Wrapper
[OK] Connection to memcached is ok