Poller.php hangs, rrdcached not creating new directores and files

Help
1

Hello,

I’m having a problem with rrdcached not creating new rrd files for devices, for example a new polling item like a new mount point, and also not creating directories and rrd files for new devices.

This is making the polling hangs on the device, never finishes polling, and is giving me some red graphs.

For example, if I add a device, the folder for this device is not created and the same happens with the rrd files for this device.

My RRDCached service is configured as below:

/sbin/rrdcached -l server.ip:42217 -l server.ip:42218 -w 600 -z 300 -f 1800 -s librenms -U librenms -G librenms -B -j /var/tmp -t 8 -F -b /opt/rrd/

The dir /opt/rrd is located in the same machine as the rrdcached service and has permission for the user librenms.

I run the poller.php for this device with the debug and verbose parameters, and I can see that it tries to read the inexistent rrd file, then it sends a command to create and also update, but the file is never created.

0% RRD[last az-librerrd/storage-hrstorage-_run_user_1000.rrd  --daemon server.ip:42217]
RRDtool Output: 0
ERROR: rrdcached: RRD Error: opening '/opt/rrd/az-librerrd/storage-hrstorage-_run_user_1000.rrd': No such file or directory
RRD[create az-librerrd/storage-hrstorage-_run_user_1000.rrd --step 600 DS:used:GAUGE:1200:0:U DS:free:GAUGE:1200:0:U  RRA:AVERAGE:0.5:1:2016 RRA:AVERAGE:0.5:6:1440 RRA:AVERAGE:0.5:24:1440 RRA:AVERAGE:0.5:288:1440  RRA:MIN:0.5:1:720 RRA:MIN:0.5:6:1440     RRA:MIN:0.5:24:775     RRA:MIN:0.5:288:797  RRA:MAX:0.5:1:720 RRA:MAX:0.5:6:1440     RRA:MAX:0.5:24:775     RRA:MAX:0.5:288:797  RRA:LAST:0.5:1:1440  -O --daemon server.ip:42217]
RRDtool Output: OK u:0.00 s:0.01 r:2.38
RRD[update az-librerrd/storage-hrstorage-_run_user_1000.rrd N:0:3372097536 --daemon server.ip:42217]
RRDtool Output: SQL[UPDATE `storage` set `storage_used` ='0',`storage_free` ='3372097536',`storage_size` ='3372097536',`storage_units` ='4096',`storage_perc` ='0' WHERE `storage_id` = '8258']

The poller hangs and the last thing show in screen before I press CTRL+C to stop the process is:

./poller.php 1848 2018-01-24 16:38:37 - 1 devices polled in 3.385 secs
SNMP [56/1.17s]: Get[36/0.76s] Getnext[0/0.00s] Walk[20/0.41s]
MySQL [105/0.75s]: Cell[12/0.01s] Row[16/0.02s] Rows[33/0.03s] Column[3/0.00s] Update[37/0.60s] Insert[3/0.07s] Delete[1/0.02s]
RRD [173/0.06s]: Update[86/0.01s] Create [1/0.00s] Other[86/0.06s]

On the RRDCached service log I can see the line below:

rrdcreate request for /opt/rrd/az-librerrd/storage-hrstorage-_run_user_1000.rrd

For new devices the folder /opt/rrd/device-name is not even created.

My version is up to date:

====================================
Component | Version
--------- | -------
LibreNMS  | 1.33-387-g0a6f113
DB Schema | 231
PHP       | 7.1.11
MySQL     | 5.5.56-MariaDB
RRDTool   | 1.6.0
SNMP      | NET-SNMP 5.7.2
====================================
2

I tried to run the rrdtool create command manually, to see what wold happen and it is this command that it is hanging.

For example, if I ran the following command, the first create on poller.php:

rrdtool create device-name/uptime.rrd --step 600 DS:uptime:GAUGE:1200:0:U  RRA:AVERAGE:0.5:1:2016 RRA:AVERAGE:0.5:6:1440 RRA:AVERAGE:0.5:24:1440 RRA:AVERAGE:0.5:288:1440  RRA:MIN:0.5:1:720 RRA:MIN:0.5:6:1440 RRA:MIN:0.5:24:775     RRA:MIN:0.5:288:797  RRA:MAX:0.5:1:720 RRA:MAX:0.5:6:1440     RRA:MAX:0.5:24:775     RRA:MAX:0.5:288:797  RRA:LAST:0.5:1:1440  -O --daemon server-ip:42218

It will just hang and I will have the following line in the rrdcached server log:

Jan 24 18:14:24 rrd-server rrdcached[116310]: rrdcreate request for /opt/rrd/device-name/uptime.rrd

But the file is not created.

3

Hello,

Anyone has any idea how to solve this? I am not able to add new devices or new polling itens in existing devices like ports and mount points, rrdcached is not creating the files.

4

Solved on Discord, the problem was caused by a missing parameter on the rrdcached, the -R parameter.

2 Likes