Distributed poller not joining cluster

I have a distributed environment composed of 2 servers: A server with all the elements, to be used as a WebUI/DB/RRD server and another one to be used only as a poller.
This is how the config.php looks like:
Server 1:

### Database config
$config['db_host'] = 'localhost';
$config['db_port'] = '3306';
$config['db_user'] = 'librenms';
$config['db_pass'] = 'password';
$config['db_name'] = 'librenms';
$config['db_socket'] = '/database/mariadb/mysql/mysql.sock';

// Distributed Poller-Settings
$config['distributed_poller']                            = true;
// optional: defaults to hostname
$config['distributed_poller_name']                      = 'server1';
$config['distributed_poller_group']                      = 0;
$config['distributed_poller_memcached_host']             = '10.0.0.51';
$config['distributed_poller_memcached_port']             = '11211';

// This is the user LibreNMS will run ad
//Please ensure this user is created and has the correct permissions to your install
$config['user'] = 'librenms';
$config['rrdtool'] = '/database/rrdtool-1.7.2/bin/rrdtool';
$config['rrdtool_version'] = '1.7.2';
$config['rrd_dir'] = '/database/rrdtool-1.7.2/libredata';
$config['rrdcached'] = 'server1:42217';
# $config['rrdcached'] = "unix:/database/rrdtool-1.7.2/rrdcached.sock";

Server 2:

// Distributed Poller-Settings
$config['distributed_poller']                            = true;
// optional: defaults to hostname
$config['distributed_poller_name']                      = 'server2';
$config['distributed_poller_group']                      = 1;
$config['distributed_poller_memcached_host']             = '10.0.0.51';
$config['distributed_poller_memcached_port']             = '11211';
// This is the user LibreNMS will run ad
//Please ensure this user is created and has the correct permissions to your install
$config['user'] = 'librenms';
$config['rrdcached'] = "10.0.0.51:42217";
$config['rrdtool'] = '/database/rrdtool-1.7.2/bin/rrdtool';
$config['db_host'] = 'server1';
$config['db_port'] = '3306';
$config['db_user'] = 'librenms';
$config['db_pass'] = 'password';
$config['db_name'] = 'librenms';

Is this enough information for Libre to understand that this is a distributed setup?

Validate looks good on both:
Server1:

sh: line 1: 117976 Killed                  sudo -K > /dev/null 2>&1
sh: line 1: 117994 Killed                  sudo -K > /dev/null 2>&1
====================================
Component | Version
--------- | -------
LibreNMS  | 1.67-11-g114df5aa8
DB Schema | 2020_08_28_212054_drop_uptime_column_outages (173)
PHP       | 7.2.33
Python    | 3.6.8
MySQL     | 5.5.65-MariaDB
RRDTool   | 1.7.2
SNMP      | NET-SNMP 5.7.2
====================================

[OK]    Composer Version: 1.10.12
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[WARN]  IPv6 is disabled on your server, you will not be able to add IPv6 devices.

Server2:

sh: line 1: 28831 Killed                  sudo -K > /dev/null 2>&1
sh: line 1: 28844 Killed                  sudo -K > /dev/null 2>&1
====================================
Component | Version
--------- | -------
LibreNMS  | 1.67-11-g114df5aa8
DB Schema | 2020_08_28_212054_drop_uptime_column_outages (173)
PHP       | 7.3.22
Python    | 3.6.8
MySQL     | 5.5.65-MariaDB
RRDTool   | 1.7.2
SNMP      | NET-SNMP 5.7.2
====================================

[OK]    Composer Version: 1.10.12
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[WARN]  IPv6 is disabled on your server, you will not be able to add IPv6 devices.

Check the contents of the .env files on both nodes

Ah I see now that you are running a single poller in the end, then it’s not distributed polling anyway if the pollers just doesn’t have the frontend.

If you run multiple pollers on the same group for example, then it’s distributed polling. You can just leave that part out of your config and make sure both servers point to the same rrdcached server.

Now I understood that distributed polling would be the case if I had both pollers in group 0, balancing the load. But that is not the case. What I want at this point is 2 independent pollers with only RRD/Frontend. Please find below the .env:

Server1:

APP_KEY=base64:xxx

DB_HOST=server1
DB_DATABASE=librenms
DB_USERNAME=librenms
DB_PASSWORD=password

#APP_URL=
NODE_ID=5f520982290d2
DB_PORT=3306
LIBRENMS_USER=librenms
REDIS_HOST=10.0.0.51
REDIS_PORT=6379
# OR
# REDIS_SENTINEL=192.0.2.1:26379
# REDIS_SENTINEL_SERVICE=myservice

Server2:

APP_KEY=base64:xxx
DB_HOST=localhost
DB_DATABASE=librenms
DB_USERNAME=librenms
DB_PASSWORD=password

#APP_URL=
NODE_ID=5f51ff0da3811
DB_PORT=3306
DB_SOCKET=/database/mariadb/mysql/mysql.sock
LIBRENMS_USER=librenms
REDIS_HOST=10.0.0.51
REDIS_PORT=6379
# OR
# REDIS_SENTINEL=192.0.2.1:26379
# REDIS_SENTINEL_SERVICE=myservice

REDIS_DB=0
#REDIS_PASSWORD=
#REDIS_TIMEOUT=60

But you mentioned: “A server with all the elements, to be used as a WebUI/DB/RRD server and another one to be used only as a poller.” so only one would be polling?

Now yes, though I would like to have the Frontend one “able” to poll as well. But in the future, I will certainly add new pollers.

Yea ok, so the server running frontend could also be polling. Clear :slight_smile:

Looking at the configs I don’t see anything odd, did you try to start the services/cronjobs and see what the output from them is?

Hi Elias,

I found the issue. Redis was not running properly on server1. After going through the configuration steps, the new poller was registered.
I moved the devices from one poller_group to another one using MySQL queries and it worked :slight_smile: Thanks for helping me out.
By the way, I only found these Redis errors in /var/log/messages.

Ah good to hear! Usually the poller logs should show if it can’t connect to redis. Are you using the service or cronjobs?

Service. It is really weird, but the logs/librenms.log file hasn’t even been created on server2.

Check if you see anything with systemctl status librenms or via journalctl, the poller logs should be there.