Distributed Poller Setup - About ready to give up!

Whiskey-Water · 22 February 2022 16:51

Hello I also put this on Reddit in case anyone is watching that.

Trying my best to get this going. Videos are old, documentation is old. Seems like I am playing a game of whack a mole.

First on the secondary poller I was getting rrdcached errors:

RRD[last 192.168.100.5/port-id44.rrd --daemon unix:/var/run/rrdcached.sock]

RRDtool Output: 1645212968

ERROR: Unable to connect to rrdcached: No such file or directory

FAIL] /var/run/rrdcached.sock doesn’t appear to exist, rrdcached test failed

So I edited the config.php file on the secondary poller to point it to primary DB/Poller $config[‘rrdcached’] = “192.168.100.204:42217”;

This made the polling errors go away however now when I do a ./validate.php -g distributedpoller on the secondary poller it says this

librenms@libre-poller:~$ ./validate.php -g distributedpoller

====================================

Component | Version

--------- | -------

LibreNMS | 22.2.1-6-g703745d0e

DB Schema | 2022_02_03_164059_increase_auth_id_length (235)

PHP | 7.4.3

Python | 3.8.10

MySQL | 10.3.32-MariaDB-0ubuntu0.20.04.1

RRDTool | 1.7.2

SNMP | 5.8

====================================

[OK] Composer Version: 2.2.6

[OK] Dependencies up-to-date.

Checking distributedpoller: FAIL

[INFO] Detected Dispatcher Service

[FAIL] Dispatcher service is enabled on your cluster, but not in use on this node

AND my graphs are not updating for any device being polled by the secondary poller.

The primary poller I also edited the config.php with the line $config[‘rrdcached’] = “192.168.100.204:42217”;

because you cannot update it under Pollers in the GUI as it is greyed out. The video I watched showed it being manually updated in the GUI but that isn’t an option any longer it appears. Also in that video it said the secondary poller would read the config off the main poller for rrdcached and be good to go however it does not as I get the RRD errors previously mentioned.

I am on the cusp here but again is seems like whack a mole. Been working on this for days. Please help!

Primary Poller/DB

librenms@librenms:~$ ./validate.php

====================================

Component | Version

--------- | -------

LibreNMS | 22.2.1-6-g703745d0e

DB Schema | 2022_02_03_164059_increase_auth_id_length (235)

PHP | 7.4.3

Python | 3.8.10

MySQL | 10.3.32-MariaDB-0ubuntu0.20.04.1

RRDTool | 1.7.2

SNMP | 5.8

====================================

[OK] Composer Version: 2.2.6

[OK] Dependencies up-to-date.

[OK] Database connection successful

[OK] Database schema correct

[WARN] Dispatcher Service has been used in your cluster, but not recently. It may take up to 5 minutes to register.

[INFO] Detected Python Wrapper

[OK] Connection to memcached is ok

Secondary Poller

librenms@libre-poller:~$ ./validate.php

====================================

Component | Version

--------- | -------

LibreNMS | 22.2.1-6-g703745d0e

DB Schema | 2022_02_03_164059_increase_auth_id_length (235)

PHP | 7.4.3

Python | 3.8.10

MySQL | 10.3.32-MariaDB-0ubuntu0.20.04.1

RRDTool | 1.7.2

SNMP | 5.8

====================================

[OK] Composer Version: 2.2.6

[OK] Dependencies up-to-date.

[OK] Database connection successful

[OK] Database schema correct

[WARN] Dispatcher Service has been used in your cluster, but not recently. It may take up to 5 minutes to register.

[INFO] Detected Python Wrapper

[OK] Connection to memcached is ok

librenms@libre-poller:~$

Secondary Poller

librenms@libre-poller:~$ ./validate.php -g distributedpoller

====================================

Component | Version

--------- | -------

LibreNMS | 22.2.1-6-g703745d0e

DB Schema | 2022_02_03_164059_increase_auth_id_length (235)

PHP | 7.4.3

Python | 3.8.10

MySQL | 10.3.32-MariaDB-0ubuntu0.20.04.1

RRDTool | 1.7.2

SNMP | 5.8

====================================

[OK] Composer Version: 2.2.6

[OK] Dependencies up-to-date.

Checking distributedpoller: FAIL

[INFO] Detected Dispatcher Service

[FAIL] Dispatcher service is enabled on your cluster, but not in use on this node

THANK YOU!

paulierco · 22 February 2022 17:33

Can you share the content of the file: /etc/default/rrdcached ?

Whiskey-Water · 22 February 2022 19:44

Hello and Thanks for responding.

Primary =
librenms@librenms:/etc/default$ cat rrdcached
DAEMON=/usr/bin/rrdcached
DAEMON_USER=librenms
DAEMON_GROUP=librenms
WRITE_THREADS=4
WRITE_TIMEOUT=1800
WRITE_JITTER=1800
BASE_PATH=/opt/librenms/rrd/
JOURNAL_PATH=/var/lib/rrdcached/journal/
PIDFILE=/run/rrdcached.pid
SOCKFILE=/run/rrdcached.sock
SOCKGROUP=librenms
BASE_OPTIONS="-B -F -R -l 0:42217"

Secondary = does not have rrdcached therefore no file.

paulierco · 23 February 2022 10:46

First of all,

I don’t see this:

NETWORK_OPTIONS=“-L”

Second, i’m not sure about:

BASE_OPTIONS=“-B -F -R -l 0:42217”

Could you please try to replace it with:

BASE_OPTIONS=“-B -F -R”

Do not forget to restart rrdcached after this change. Also make sure service is running after restart.

Whiskey-Water · 23 February 2022 14:09

Good morning - I am following this video verbatim and he explains the reasons for the config.

I believe the -L is now the same as -l
That is the problem nothing seems to be current. There is a ton of old documentation but nothing about how these current revisions are supposed to work. Maybe it isn’t even possible how get distributed polling to work with the current revisions? I haven’t seen anyone do it. The most current revision I have found is this video and another gentleman on reddit who used redis and memcache which I am also now doing but mine simply does not work

paulierco · 23 February 2022 14:15

So that might be changed now due to new versions of rrdcached. Not sure if that will work for new version. I checked now but on the docs is something that I mentioned.

Have to tried to do that I proposed ?
Also, please paste the output of:

sudo systemctl status rrdcached

Whiskey-Water · 23 February 2022 16:05

So I rebuilt my secondary poller and I believe I originally missed a couple of things on that secondary poller/
sudo pip3 install -r requirements.txt
sudo mv config.php.default config.php
nano config.php

$config[‘distributed_poller’] = true; # Set to true to enable distributed polling
$config[‘distributed_poller_name’] = php_uname(‘n’); # Uniquely identifies the poller instance
$config[‘distributed_poller_group’] = 2; # Which group to poll

sudo cp /opt/librenms/misc/librenms.service /etc/systemd/system/librenms.service && systemctl enable --now librenms.service
systemctl status librenms

Doing all of this appears to have taken care of it!! I never touched changed anything on the primary which would be what you are referring to.

I do not believe I need the distributed poller stuff in the secondary poller config.php file as it is now availble in the GUI but I am going to leave it for now.

Thanks for chiming in!

system · 24 May 2022 16:05

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.