RRD Graph not being created on new distributed pollers

Help
1

Hi,
I am trying to migrate devices to existing single poller to a distributed system. I have the following setup:

1 box: rrdcached and memcache
1 box: new poller
1 box: old database.

I updated the database record to move 1 device from poller group 1 to pollergroup 4. (i.e. the new poller I want to use)

Excerpts from config.php:
$config[‘distributed_poller_group’] = ‘4’;
$config[‘distributed_poller_memcached_host’] = ‘X.X.X.X’;
$config[‘distributed_poller_memcached_port’] = ‘11211’;
$config[‘rrdcached’] = “X.X.X.X:42217”;

Output of Poller.php
./poller.php -h X.X.X.X

LibreNMS Poller

Starting polling run:

Hostname: X.X.X.X

Device ID: 782

OS: cumulus

(unix)

End Alerts

SNMP [40/5.65s]: Get[19/0.83s] Getnext[1/0.03s] Walk[20/4.79s]

MySQL [450/0.24s]: Cell[95/0.03s] Row[-95/-0.03s] Rows[58/0.04s] Column[1/0.00s] Update[389/0.20s] Insert[2/0.00s] Delete[0/0.00s]

RRD [604/0.05s]: Update[302/0.01s] Create [0/0.00s] Other[302/0.04s]

Output of Poller wrapper (There is only 1 test device on this poller)
./poller-wrapper.py

Registered as Master

INFO: starting the poller at 2019-05-13 10:38:44 with 16 threads, slowest devices first

INFO: worker Thread-2 finished device 782 in 10 seconds

INFO: poller-wrapper polled 1 devices in 11 seconds with 16 workers

Output of validate.php
./validate.php

Component Version
LibreNMS 1.51-14-g554e22d
DB Schema 2019_02_10_220000_add_dates_to_fdb (390)
PHP 7.2.16
MySQL 5.7.14-8-log
RRDTool 1.7.1
SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 1.8.5
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[FAIL] Discovery has not completed in the last 24 hours.
[FIX]:
Check the cron job to make sure it is running and using discovery-wrapper.py

Output of validate.php distributedpoller
bash-4.2$ ./validate.php -g distributedpoller

Component Version
LibreNMS 1.51-14-g554e22d
DB Schema 2019_02_10_220000_add_dates_to_fdb (390)
PHP 7.2.16
MySQL 5.7.14-8-log
RRDTool 1.7.1
SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 1.8.5
[OK] Dependencies up-to-date.
Checking distributedpoller: OK
[OK] Connection to memcached is ok
bash-4.2$

RRD Command on the GUI
Showing the RRD command on the GUI throws an error:
ERROR: rrdcached:X.X.X.X:42217: Unknown Command: FETCH

RRD Cache
Is running:
sudo systemctl status rrdcached
rrdcached.service - Data caching daemon for rrdtool
Loaded: loaded (/etc/systemd/system/rrdcached.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-05-13 09:16:59 CDT; 1h 35min ago
Main PID: 429251 (rrdcached)
CGroup: /system.slice/rrdcached.service

And is listening:
sudo netstat -anp | grep 42217
tcp 0 0 10.254.99.120:42217 0.0.0.0:* LISTEN 429251/rrdcached

When I did the migration of the device, it created a new folder in /opt/librenms/rrd with the IP address, but that folder is still empty. What else can I try next to get this working?

2

I got a little farther. It seems it’s rrdcache again! The service is not starting correctly.

sudo /usr/bin/rrdcached -w 1800 -z 1800 -f 3600 -s librenms -U librenms -G librenms -B -R -j /var/tmp -l X.X.X.X -t 4 -F -b /opt/librenms/rrd/
rrdcached: can’t create pid file ‘/opt/rrdtool-1.7.1/var/run/rrdcached.pid’ (File exists)
FATAL: Another rrdcached daemon is running?? (pid 156484)

3

Maybe you already have it running? Or a zombie pid file?