Distributed Cannot Consistently Poll Devices

Help
1

We are running into an issue where when we enable distributed pollers, previously discovered devices no longer show as online. We’ve verified the hosts can communicate with the devices by manually performing snmpwalk, ping and ssh from the pollers to the network devices…

The IPs of the pollers are in the device’s ACL, and the pollers can all talk to the central MariaDB, rrdcached and memcached instances.

Initial deployment
root@host:/opt/librenms# ./validate.php

Component Version
LibreNMS 1.58.1-29-g37d6a86c5
DB Schema 2019_10_03_211702_serialize_config (147)
PHP 7.2.24-0ubuntu0.18.04.1
MySQL 10.1.43-MariaDB-0ubuntu0.18.04.1
RRDTool 1.7.0
SNMP NET-SNMP 5.7.3

====================================

[OK] Composer Version: 1.9.1
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[WARN] Some devices have not been polled in the last 5 minutes. You may have performance issues.
…

distributed poller
root@host:/opt/librenms# ./validate.php

Component Version
LibreNMS 1.58.1-29-g37d6a86c5
DB Schema 2019_10_03_211702_serialize_config (147)
PHP 7.2.24-0ubuntu0.18.04.1
MySQL 10.1.43-MariaDB-0ubuntu0.18.04.1
RRDTool 1.7.0
SNMP NET-SNMP 5.7.3

====================================

[OK] Composer Version: 1.9.1
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[WARN] Some devices have not been polled in the last 5 minutes. You may have performance issues.
…

2

Look at the poller log and see what is taking the longest to pull.

3

We had a couple devices regularly polling over 250s and have removed them from Libre for the time being. But we are still seeing devices on our distributed pollers not always being polled. Most devices are polling fine on the distributed pollers.

Here is the sanitized config being used on our distributed pollers.

<?php
## Have a look in defaults.inc.php for examples of settings you can set here. DO NOT EDIT defaults.inc.php!

### Database config
$config['db_host'] = 'DB Server';
$config['db_port'] = '3306';
$config['db_user'] = 'librenms';
$config['db_pass'] = 'DB PASSWORD';
$config['db_name'] = 'librenms';
$config['db_socket'] = '';

// This is the user LibreNMS will run as
//Please ensure this user is created and has the correct permissions to your install
$config['user'] = 'librenms';

### Default community
$config['snmp']['community'] = array(SNMP STRINGS);

### Authentication Model
$config['auth_mechanism'] = "mysql"; # default, other options: ldap, http-auth

### List of RFC1918 networks to allow scanning-based discovery
$config['nets'][] = "10.0.0.0/8";


#Distributed Poller Setup
$config['distributed_poller_name'] = php_uname('n');
$config['distributed_poller_group'] = '0';
$config['distributed_poller_memcached_host'] = 'MEMCACHED Server';
$config['distributed_poller_memcached_port'] = 11211;
$config['distributed_poller'] = true;
$config['rrdtool_version'] = '1.7.0';
$config['rrdcached'] = 'RRDCACHED Server:42217';
#$config['update'] = 0;

# Disable/Enable Pollers
$config['poller_modules']['unix-agent']                  = false;
$config['poller_modules']['os']                          = true;
$config['poller_modules']['ipmi']                        = true;
$config['poller_modules']['sensors']                     = true;
$config['poller_modules']['processors']                  = true;
$config['poller_modules']['mempools']                    = true;
$config['poller_modules']['storage']                     = true;
$config['poller_modules']['netstats']                    = true;
$config['poller_modules']['hr-mib']                      = true;
$config['poller_modules']['ucd-mib']                     = true;
$config['poller_modules']['ipSystemStats']               = true;
$config['poller_modules']['ports']                       = true;
$config['poller_modules']['nac']                         = false;
$config['poller_modules']['bgp-peers']                   = false;
$config['poller_modules']['junose-atm-vp']               = false;
$config['poller_modules']['toner']                       = false;
$config['poller_modules']['ucd-diskio']                  = true;
$config['poller_modules']['wireless']                    = false;
$config['poller_modules']['ospf']                        = true;
$config['poller_modules']['cisco-ipsec-flow-monitor']    = false;
$config['poller_modules']['cisco-remote-access-monitor'] = false;
$config['poller_modules']['cisco-cef']                   = false;
$config['poller_modules']['cisco-sla']                   = false;
$config['poller_modules']['cisco-mac-accounting']        = false;
$config['poller_modules']['cipsec-tunnels']              = false;
$config['poller_modules']['cisco-ace-loadbalancer']      = false;
$config['poller_modules']['cisco-ace-serverfarms']       = false;
$config['poller_modules']['cisco-asa-firewall']          = false;
$config['poller_modules']['cisco-voice']                 = false;
$config['poller_modules']['cisco-cbqos']                 = false;
$config['poller_modules']['cisco-otv']                   = false;
$config['poller_modules']['cisco-vpdn']                  = false;
$config['poller_modules']['netscaler-vsvr']              = false;
$config['poller_modules']['aruba-controller']            = false;
$config['poller_modules']['entity-physical']             = true;
$config['poller_modules']['entity-state']                = false;
$config['poller_modules']['applications']                = true;
$config['poller_modules']['mib']                         = false;
$config['poller_modules']['stp']                         = true;
$config['poller_modules']['ntp']                         = true;
$config['poller_modules']['services']                    = true;
$config['poller_modules']['loadbalancers']               = false;
$config['poller_modules']['mef']                         = false;

# fping tuning
$config['fping_options']['timeout'] = 300;
$config['fping_options']['count']   = 1;
$config['fping_options']['interval'] = 300;