Distributed poller configured with redis, warns about memcached

[danny]

Can anyone clear up some documentation confusion please? Looking at:

Scaling LibreNMS - LibreNMS Docs

It states “Pick one of the following setups, do not use all of them at the same time.”, so we’ve configured two servers, one of which is our DB/Redis/Web server, the other is just our Poller. We selected redis because documentation states “preferred”

Config files look like:

vim config.php

$config['distributed_poller']                       = true;
$config['distributed_poller_group']                 = 1;

vim .env

REDIS_HOST=192.168.10.5
REDIS_PORT=6379
CACHE_DRIVER=redis

However, the log files on the poller give the following warning:
tail /opt/librenms/logs/poller_wrapper.log
2022-02-09 18:25:05,391 :: WARNING :: Could not connect to memcached, disabling distributed service checks.

./validate.php also fails because there is no memcached:
[FAIL] We could not get memcached stats, it is possible that we cannot connect to your memcached server, please check

So, is the “Pick one” wrong or misleading? Even if we use the preferred Redis, we still need memcached as well?

[Kevin_Krumm]

check out - LibreNMS Distributed Polling - YouTube

[danny]

Thanks, but that doesn’t answer my question. That video is more of a tutorial. I’ve already followed various tutorials. None of which explain the issue as to why are the logs and ./validate.php are complaining about memcached, if memcached is not configured anywhere. Per the documentation, and that tutorial, all references to memcached have been commented out.

[laf]

Looks like you are running the wrapper scripts, you will need to switch to the Dispatcher service (Dispatcher Service (RC) - LibreNMS Docs)

[danny]

Hey laf,

Thanks for the quick reply. I thought about that…and I was going to get to that eventually. However, would that explain why ./validate.php is failing?

====================================
Component | Version
--------- | -------
LibreNMS  | 22.1.0-54-gb9e55bc46
DB Schema | 2021_12_02_113537_ports_stp_designated_cost_change_to_int (234)
PHP       | 7.4.19
Python    | 3.6.8
MySQL     | 10.6.5-MariaDB-log
RRDTool   | 1.7.0
SNMP      | 5.8
====================================

[OK]    Composer Version: 2.2.6
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[INFO]  Detected Python Wrapper
[FAIL]  We could not get memcached stats, it is possible that we cannot connect to your memcached server, please check

I don’t have memcached configured anywhere (that I can find), so why is that failing?

[laf]

Because you are using the wrapper scripts which depend on memcached so validate is checking for that. Switch to the dispatcher and the issue will go away.

[Kevin_Krumm]

it was in the video

[danny]

Nope, still missing something.

Dispatcher setup straight forward and easy, but validate is sill failing. I can see redis MONITOR scrolling by with both IPs (web server and poller) and /opt/librenms/logs/librenms.log shows the normal:

/opt/librenms/poller.php 1 2022-02-10 12:55:53 - 1 devices polled in 5.100 secs
/opt/librenms/poller.php 132 2022-02-10 12:55:53 - 1 devices polled in 4.999 secs
/opt/librenms/poller.php 160 2022-02-10 12:55:53 - 1 devices polled in 4.470 secs
/opt/librenms/poller.php 80 2022-02-10 12:55:53 - 1 devices polled in 3.961 secs
…etc.

And syslog shows new:
Feb 10 12:53:21 librenms-02 librenms-service.py[2045997]: Discovery_1-5(INFO):Completed discovery run for 299 (1) in 19.99s
Feb 10 12:53:21 librenms-02 librenms-service.py[2045997]: Discovery_1-5(INFO):Discovering device 320
Feb 10 12:53:22 librenms-02 librenms-service.py[2045997]: Discovery_1-6(INFO):Completed discovery run for 295 (1) in 20.32s
Feb 10 12:53:22 librenms-02 librenms-service.py[2045997]: Discovery_1-6(INFO):Discovering device 318
Feb 10 12:53:22 librenms-02 librenms-service.py[2045997]: Discovery_1-15(INFO):Completed discovery run for 276 (1) in 21.38s
…etc.

But still getting the same complaint about memcached.

[laf]

Screenshot your poller settings page /settings/poller/distributed

[danny]

[danny]

[laf]

I’m at a loss

Post a copy of validate again

[danny]

@laf

I’m at a loss

Welcome to my world

====================================
Component | Version
--------- | -------
LibreNMS  | 22.1.0-55-ga49ee59c0
DB Schema | 2021_12_02_113537_ports_stp_designated_cost_change_to_int (234)
PHP       | 7.4.19
Python    | 3.6.8
MySQL     | 10.6.5-MariaDB-log
RRDTool   | 1.7.0
SNMP      | 5.8
====================================

[OK]    Composer Version: 2.2.6
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[INFO]  Detected Python Wrapper
[FAIL]  We could not get memcached stats, it is possible that we cannot connect to your memcached server, please check

[danny]

Nevermind. That worked!

I was running validate too quickly. I thought that validate would pass per node…but you have to update ALL nodes at the same time, and then validate passes for everything. Interesting, but good to know.

Thank you both for all the quick help, everything is good to go!