Hello LibreNMS community,
I have a distrubuted poller setup running librenms-dispatcher and this works wonderfully to a galera backend database and the pollers are connected currently to a single redis instance.
I have an almost symmetric replica in my lab setup to test this. I am attempting to make the redis highly available, so I followed the instructions at Dispatcher Service (RC) - LibreNMS Docs and replaced my .env references for REDIS (snipping from my Lab configuration contains two active and working distributed pollers, but with this poller currently stopped).
##
## Standalone
##
REDIS_HOST=10.101.101.21
REDIS_PORT=6379
REDIS_DB=0
REDIS_TIMEOUT=60
with the Sentinel variant as follows:
##
## Sentinel
##
REDIS_SENTINEL=10.101.101.21:26379,10.101.101.22:26379,10.101.101.23:26379
REDIS_SENTINEL_SERVICE=redis-primary
I realise there is no authentication set up, I took that out to troubleshoot if the auth was causing problems but end up with exactly the same results. The principle issue I am having is that if I run a validate.php, it outputs that redis is unavailable. Performing a packet capture on the 3 nodes above, there is zero attempt for the libre node being validated attempting to connect at all to the sentinel nodes.
I have verified that I can talk to the sentinel TCP port, so it is not a routing / firewall related issue, it simply doesnt make an attempt to connect as if the configuration in .env may not be being read.
Trying to work around this to see if different things work, thinking maybe it was a missing:
CACHE_DRIVER=redis
However, if I re-run validate.php, as I have ommitted the:
REDIS_HOST=10.101.101.21
The validate script attempts to connect on localhost:6379 and errors out (principally because redis-server is not present on the libre node being validated). I cannot see anywhere else in the documentation for an alternative for CACHE_DRIVER apart from memcached, so I believe I am missing something somewhere. I could put in the current master for the redis, but there is little point I think in doing that as it will permanetly use that redis-server, thwarting the highly available functionality. I believe that LibreNMS should be connecting to at least one of the sentinels and runing the likes of:
SENTINEL get-master-addr-by-name redis-primary
which will return the current master and port, and then LibreNMS should connect via the redis port and do its tests. If I run against any one of the sentinels manually using redis-cli, I do observe the IP and port of the master. If I deliberatly down the current master, I see a new IP for the slave replica… so I think I have my intial redis / sentinel cluster working ok.
So my question surrounds my approach to using Redis Sentinel in terms of LibreNMS’s .env config. Is this fully supported in LibreNMS, or should I revert back to a regular Redis config using a static host and use the likes of haproxy / sentinel / redis to direct to the master redis node?
Thanks for reading