Enabling Distributed polling broke graphs

Tags: #<Tag:0x00007fd888d184a0>

Hello,

I have 2 server setup

Server 1

  • Web
  • DB
  • RRDcached
  • Memcached
    -poller

Server 2

  • poller

I want both servers to poll all devices in group 0. My configs on each device look like this.

Server1

$config['user'] = 'librenms';
$config['rrdcached'] = "server1_IP:42217";
$config['distributed_poller'] = true;
$config['rrdtool_version'] = '1.7.2';
$config['distributed_poller_name'] = "mia-master";
$config['distributed_poller_group'] = 0;
$config['auth_mechanism'] = "mysql"; # default, other options: ldap, http-auth
$config['rrd_purge'] = 0;
$config['enable_billing'] = 1;
$config['show_services'] = 1;

Server2
$config[‘user’] = ‘librenms’;

$config['distributed_poller_name']           = lhrpoller;
$config['distributed_poller_group']          = '0';
$config['distributed_poller_memcached_host'] = "server1_IP";
$config['distributed_poller_memcached_port'] = 11211;
$config['distributed_poller']                = true;
$config['rrdcached']                         = "server1_IP:42217";

My .env files look li,e

server1
APP_KEY=base64:APP_KEY

DB_HOST=localhost
DB_DATABASE=librenms
DB_USERNAME=libre_user_db
DB_PASSWORD=libre_pass_db

NODE_ID=SERVER1_NODEID
LIBRENMS_USER=librenms
CACHE_DRIVER=memcached

server2
APP_KEY=base64:APP_KEY

DB_HOST=server1_IP
DB_DATABASE=librenms
DB_USERNAME=libre_pass_db
DB_PASSWORD=libre_pass_db

NODE_ID=SERVER2_NODEID
LIBRENMS_USER=librenms
CACHE_DRIVER=memcached

Issues I’m seeing

  • Graphing does not work anywhere
  • The pollers in the web interface are both red, my server2 poler shows no name, no last poll date, and no devices polled (number of devices). My primary poller only shows 1 device monitored.
  • Devices show the last discovery as right before i enabled distributed polling

Troubleshooting I’ve tried

  • run validate.php on both servers, and with the -distributedpoller flag, returns no errors.
  • I run poller.php on server1, it runs through all my devices, and I see it creating rrd files on the TCP socket, and I see it updating the database.
    – Output from forced poll

SNMP [39/65.85s]: Get[15/1.53s] Getnext[4/0.40s] Walk[20/63.92s]
MySQL [683/1.73s]: Cell[8/0.01s] Row[-8/-0.01s] Rows[20/0.08s] Column[1/0.00s] Update[661/1.64s] Insert[1/0.00s] Delete[0/0.00s]
RRD [894/0.51s]: Other[447/0.36s] Update[447/0.16s]

  • I ran the .dist-poller.php -r on my secondary server which is when it added a 2nd poller, but as mentioned no name, and poller date is all 0’s.

RRD Version - RRDtool 1.7.2
RRDCached - RRDCacheD 1.7.2

RRDcached config

DAEMON=/usr/bin/rrdcached
WRITE_TIMEOUT=1800
WRITE_JITTER=1800
WRITE_THREADS=4
BASE_PATH=/opt/librenms/rrd/
JOURNAL_PATH=/var/lib/rrdcached/journal/
PIDFILE=/run/rrdcached.pid
SOCKFILE=/run/rrdcached.sock
SOCKGROUP=librenms
DAEMON_GROUP=librenms
DAEMON_USER=librenms
BASE_OPTIONS="-l 0:42217"
BASE_OPTIONS="$BASE_OPTIONS -R -j /var/lib/rrdcached/journal/ -F"
BASE_OPTIONS="$BASE_OPTIONS -b /opt/librenms/rrd -B"
BASE_OPTIONS="$BASE_OPTIONS -w 1800 -z 900"