RRDcached: port-id*-dot3.rrd not created nor recreated by rrdcached

czilian · 11 March 2018 11:49

RRDcached works fine with all graphs except for ports for which POE was recognized (asking myself sometimes also why). Syslog captures a "handle_request_update failure (cannot find the respective -ds3.rrd) and the consequent rrd creation also fails. I checked authorization etc, but all ok (for all other rrd’s it works anyhow). Since I don’t know the dependencies between port-id-poe.rrd and port-id*-dot3.rrd I am now stuck with my investigation.
I can cheat by creating the file port-id*-dot3.rrd manually. Then there is no error message. But the port-id*poe.rrd is also not updated anymore. Since this issue is with about 600 ports, I would be happy if I don’t have to cheat for all. Background explanation and / or a concrete fix would be highly appreciated.

Using one Server/VM as an example. Attached a syslog extract, poller extract and a list of the rrd files created.

Port eth0: Microsoft Corporation Device 0003 (2 / #247517) ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInMulticastPkts ifOutMulticastPkts ifHighSpeed dot3Duplex VLAN == Using ifDescr as ifAliasMicrosoft Corporation Device 0003
ifInOctets (92435 B) 770.29166666667 Bps 120 secs
ifOutOctets (24830 B) 206.91666666667 Bps 120 secs
ifInErrors (0 B) 0 Bps 120 secs
ifOutErrors (0 B) 0 Bps 120 secs
ifInUcastPkts (111 B) 0.925 Bps 120 secs
ifOutUcastPkts (111 B) 0.925 Bps 120 secs
ifInNUcastPkts (0 B) 0 Bps 120 secs
ifOutNUcastPkts (0 B) 0 Bps 120 secs
ifInDiscards (0 B) 0 Bps 120 secs
ifOutDiscards (0 B) 0 Bps 120 secs
ifInUnknownProtos (0 B) 0 Bps 120 secs
ifInBroadcastPkts (0 B) 0 Bps 120 secs
ifOutBroadcastPkts (0 B) 0 Bps 120 secs
ifInMulticastPkts (807 B) 6.725 Bps 120 secs
ifOutMulticastPkts (0 B) 0 Bps 120 secs
bps(6.160kbps/1.660kbps)bytes(90.270kB/24.250kB)pkts(0.930pps/0.930pps)RRD[ [0;32mlast duisburg.dcn.local/port-id247517.rrd --daemon unix:/var/run/rrdcached.sock [0m]
RRD[ [0;32mupdate duisburg.dcn.local/port-id247517.rrd N:2044332583:290092718:0:0:1392430:1336106:U:U:0:0:U:0:0:17850475:0 --daemon unix:/var/run/rrdcached.sock [0m]
RRD[ [0;32mlast duisburg.dcn.local/port-id247517-dot3.rrd --daemon unix:/var/run/rrdcached.sock [0m]
RRD[ [0;32mcreate duisburg.dcn.local/port-id247517-dot3.rrd --step 300 RRA:AVERAGE:0.5:1:2016 RRA:AVERAGE:0.5:6:1440 RRA:AVERAGE:0.5:24:1440 RRA:AVERAGE:0.5:288:1440 RRA:MIN:0.5:1:720 RRA:MIN:0.5:6:1440 RRA:MIN:0.5:24:775 RRA:MIN:0.5:288:797 RRA:MAX:0.5:1:720 RRA:MAX:0.5:6:1440 RRA:MAX:0.5:24:775 RRA:MAX:0.5:288:797 RRA:LAST:0.5:1:1440 -O --daemon unix:/var/run/rrdcached.sock [0m]
RRD[ [0;32mupdate duisburg.dcn.local/port-id247517-dot3.rrd N --daemon unix:/var/run/rrdcached.sock [0m]
EtherLike RRD[ [0;32mlast duisburg.dcn.local/port-id247517-poe.rrd --daemon unix:/var/run/rrdcached.sock [0m]
RRD[ [0;32mupdate duisburg.dcn.local/port-id247517-poe.rrd N:U:U:U:U --daemon unix:/var/run/rrdcached.sock [0m]
PoE SQL[ [0;33mUPDATE ports set poll_time =‘1520589162’,poll_prev =‘1520589042’,poll_period =‘120’,ifLastChange =‘0’,ifInOctets =‘2044332583’,ifInOctets_prev =‘2044240148’,ifInOctets_rate =‘770.29166666667’,ifInOctets_delta =‘92435’,ifOutOctets =‘290092718’,ifOutOctets_prev =‘290067888’,ifOutOctets_rate =‘206.91666666667’,ifOutOctets_delta =‘24830’,ifInErrors =‘0’,ifInErrors_prev =‘0’,ifInErrors_rate =‘0’,ifInErrors_delta =‘0’,ifOutErrors =‘0’,ifOutErrors_prev =‘0’,ifOutErrors_rate =‘0’,ifOutErrors_delta =‘0’,ifInUcastPkts =‘1392430’,ifInUcastPkts_prev =‘1392319’,ifInUcastPkts_rate =‘0.925’,ifInUcastPkts_delta =‘111’,ifOutUcastPkts =‘1336106’,ifOutUcastPkts_prev =‘1335995’,ifOutUcastPkts_rate =‘0.925’,ifOutUcastPkts_delta =‘111’ WHERE port_id = ‘247517’ [0m]
SQL[ [0;33mUPDATE ports_statistics set ifInNUcastPkts =‘0’,ifInNUcastPkts_prev =‘0’,ifInNUcastPkts_rate =‘0’,ifInNUcastPkts_delta =‘0’,ifOutNUcastPkts =‘0’,ifOutNUcastPkts_prev =‘0’,ifOutNUcastPkts_rate =‘0’,ifOutNUcastPkts_delta =‘0’,ifInDiscards =‘0’,ifInDiscards_prev =‘0’,ifInDiscards_rate =‘0’,ifInDiscards_delta =‘0’,ifOutDiscards =‘0’,ifOutDiscards_prev =‘0’,ifOutDiscards_rate =‘0’,ifOutDiscards_delta =‘0’,ifInUnknownProtos =‘0’,ifInUnknownProtos_prev =‘0’,ifInUnknownProtos_rate =‘0’,ifInUnknownProtos_delta =‘0’,ifInBroadcastPkts =‘0’,ifInBroadcastPkts_prev =‘0’,ifInBroadcastPkts_rate =‘0’,ifInBroadcastPkts_delta =‘0’,ifOutBroadcastPkts =‘0’,ifOutBroadcastPkts_prev =‘0’,ifOutBroadcastPkts_rate =‘0’,ifOutBroadcastPkts_delta =‘0’,ifInMulticastPkts =‘17850475’,ifInMulticastPkts_prev =‘17849668’,ifInMulticastPkts_rate =‘6.725’,ifInMulticastPkts_delta =‘807’,ifOutMulticastPkts =‘0’,ifOutMulticastPkts_prev =‘0’,ifOutMulticastPkts_rate =‘0’,ifOutMulticastPkts_delta =‘0’ WHERE port_id = ‘247517’ [0m]
2 updated

Runtime for poller module ‘ports’: 0.4247 seconds with 34168 bytes
SNMP: [17/0.38s] MySQL: [6/0.01s] RRD: [11/0.01s]

murrant · 11 March 2018 13:41

Run this exact command as the librenms user and report the output

czilian · 12 March 2018 01:24

Thx.

Also checked the RRDcached Base_Path again: =/opt/librenms/rrd/

murrant · 12 March 2018 02:01

I think your rrdcached isn’t running with the right options.
Run ps aux|grep rrdcached

And re-read this: https://docs.librenms.org/#Extensions/RRDCached/#setting-up-rrdcached

czilian · 12 March 2018 02:38

Looks ok to me.:

czilian@hamburg:/opt/librenms$ sudo ps aux|grep rrdcached
librenms 26925 0.5 0.2 2245368 38732 ? Ssl Mar11 9:00 /usr/bin/rrdcached -B -F -R -w 1800 -z 1800 -t 4 -b /opt/librenms/rrd/ -j /var/lib/rrdcached/journal/ -G librenms -U librenms -p /var/run/rrdcached.pid -s librenms -l unix:/var/run/rrdcached.sock
czilian 52513 0.0 0.0 14228 904 pts/3 S+ 10:31 0:00 grep --color=auto rrdcached
czilian@hamburg:/opt/librenms$

DAEMON=/usr/bin/rrdcached
DAEMON_USER=librenms
DAEMON_GROUP=librenms
WRITE_THREADS=4
WRITE_TIMEOUT=1800
WRITE_JITTER=1800
BASE_PATH=/opt/librenms/rrd/
JOURNAL_PATH=/var/lib/rrdcached/journal/
PIDFILE=/run/rrdcached.pid
SOCKFILE=/run/rrdcached.sock
SOCKGROUP=librenms
BASE_OPTIONS="-B -F -R"

czilian · 12 March 2018 03:07

Just deleted the device and recreated it. Now you can see in the log that for all rrd values it works except for the .dot3.rdd. Hence, should have nothing to do with rrdcached really.

murrant · 12 March 2018 18:21

If found your bug, empty rrd definition.

czilian · 12 March 2018 22:04

Confirmed. All .dot3.rrd are created. Syslog is at peace again.

czilian · 12 March 2018 22:05

Thank you very much murrant. You made my day. Great responsive support!