Hi everyone,
I’m running a distributed cluster with a dedicated RRDcached server. All hosts are Ubuntu 18 with the latest repo version of rrdcached, 1.7.0-1build1. I have global variable rrdtool_tune enabled, so all RRD port files should be tuned when the port speed changes, but I am seeing an issue when trying to do this over rrdcached.
After collecting polling logs, I found the rrd tune command that the poller is running, here:
RRD[tune 192.168.7.3/port-id146.rrd --maximum INOCTETS:12500000 --maximum OUTOCTETS:12500000 --maximum INERRORS:12500000 --maximum OUTERRORS:12500000 --maximum INUCASTPKTS:12500000 --maximum OUTUCASTPKTS:12500000 --maximum INNUCASTPKTS:12500000 --maximum OUTNUCASTPKTS:12500000 --maximum INDISCARDS:12500000 --maximum OUTDISCARDS:12500000 --maximum INUNKNOWNPROTOS:12500000 --maximum INBROADCASTPKTS:12500000 --maximum OUTBROADCASTPKTS:12500000 --maximum INMULTICASTPKTS:12500000 --maximum OUTMULTICASTPKTS:12500000 --daemon rrdcached.librenms.local:42217]
When running that manually from a poller host, I get a “file not found” error:
$ rrdtool tune 192.168.7.3/port-id146.rrd [...] --daemon rrdcached.librenms.local:42217
ERROR: opening '/opt/librenms/rrd/192.168.7.3/port-id146.rrd': No such file or directory
$ echo $?
1
I have confirmed via pcap that the error is coming from the rrdcached server. When I run the same command on the rrdcached server locally but over network socket, it still fails (again, confirmed via pcap over loopback).
Any suggestions? Other rrdtool commands, like rrdtool info work remotely without an issue, it seems to only be rrdtune that fails.