Distributed polling is not working

[abatapa]

Hello.
I’m struggling for two days to run the distributed polling with no luck.

Two ubuntu servers, (both 16.04, separately Librenms on both works fine)

ubuntuS1 - main server and a poller, it’s IP: 192.168.100.23
root@ubuntuS1:/opt/librenms# php validate.php

====================================

Component	Version
LibreNMS	1.30.01-84-gb132686
DB Schema	204
PHP	7.0.22-0ubuntu0.16.04.1
MySQL	10.0.31-MariaDB-0ubuntu0.16.04.2
RRDTool	1.5.5
SNMP	NET-SNMP 5.7.3

====================================

[OK] Database connection successful
[OK] Database schema correct

ubuntuHP - should be distributed poller
root@ubuntuHP:/opt/librenms# php validate.php

Component	Version
LibreNMS	1.30.01-84-gb132686
DB Schema	204
PHP	7.0.22-0ubuntu0.16.04.1
MySQL	10.0.31-MariaDB-0ubuntu0.16.04.2
RRDTool	1.5.5
SNMP	NET-SNMP 5.7.3

====================================

[OK] Database connection successful
[OK] Database schema correct
[FAIL] The poller has not run in the last 5 minutes, check the cron job
[FAIL] Discovery has never run, check the cron job
[WARN] Some devices have not been polled in the last 5 minutes.
You may have performance issues. Check your poll log and see: http://docs.librenms.org/Support/Performance/
ubuntuhp

config.php from ubuntuS1 (server, main host)
#########################

$config[‘db_host’] = ‘localhost’;
$config[‘db_port’] = ‘3306’;
$config[‘db_user’] = ‘librenms’;
$config[‘db_pass’] = ‘password’;
$config[‘db_name’] = ‘librenms’;
$config[‘db_socket’] = ‘’;
$config[‘rrdtool_version’] = ‘1.5.5’;
$config[‘user’] = ‘librenms’;
$config[‘memcached’][‘enable’] = TRUE;
$config[‘memcached’][‘host’] = “192.168.100.23”;
$config[‘memcached’][‘port’] = 11211;

$config[‘snmp’][‘community’] = array(“public”);
$config[‘show_services’] = 1;

$config[‘rrdcached’] = “192.168.100.23:42217”;
$config[‘rrd_dir’] = “/opt/librenms/rrd”;
###########

Here is config from ubuntuHP
#########################
$config[‘db_host’] = ‘localhost’;
$config[‘db_port’] = ‘3306’;
$config[‘db_user’] = ‘librenms’;
$config[‘db_pass’] = ‘password’;
$config[‘db_name’] = ‘librenms’;
$config[‘db_socket’] = ‘’;

$config[‘user’] = ‘librenms’;
$config[‘distributed_poller’] = true;
$config[‘distributed_poller_name’] = “ubuntuhp”;
$config[‘distributed_poller_group’] = ‘2’;
$config[‘distributed_poller_memcached_host’] = ‘192.168.100.23’;
$config[‘distributed_poller_memcached_port’] = 11211;
$config[‘rrdcached’] = “192.168.100.23:42217”;
$config[‘rrd_dir’] = “/opt/librenms/rrd”;
$config[‘snmp’][‘community’] = array(“public”);
###############

• rrd cached is run:
  /usr/bin/rrdcached -B -L -w 1800 -z 1800 -b /opt/librenms/rrd/ -j /var/lib/rrdcached/journal/ -G librenms -U librenms -p /var/run/rrdcached.pid -g -F
  (it’s run on foreground, i can see what is happening)
• rrdcached works OK for ubuntuS1

Device (basic localhost, ubuntuhp) is added via GUI on ubuntuHP (poller)

Problem:

When i manually run poller on ubuntuhp (the second serv which should be dist.poller)
./poller.php -h ubuntuhp -d -v

i get such output on ubunts1 on rrdcached window:

handle_request_update: stat (/opt/librenms/rrd/ubuntuhp/uptime.rrd) failed.
handle_request_update: stat (/opt/librenms/rrd/ubuntuhp/poller-perf-os.rrd) failed.
handle_request_update: stat (/opt/librenms/rrd/ubuntuhp/poller-perf-ipmi.rrd) failed.
handle_request_update: stat (/opt/librenms/rrd/ubuntuhp/poller-perf-sensors.rrd) failed.
handle_request_update: stat (/opt/librenms/rrd/ubuntuhp/poller-perf-processors.rrd) failed.
handle_request_update: stat (/opt/librenms/rrd/ubuntuhp/poller-perf-mempools.rrd) failed.
handle_request_update: stat (/opt/librenms/rrd/ubuntuhp/poller-perf-storage.rrd) failed.
handle_request_update: stat (/opt/librenms/rrd/ubuntuhp/netstats-ip.rrd) failed.
handle_request_update: stat (/opt/librenms/rrd/ubuntuhp/netstats-tcp.rrd) failed.
… and so on.

And the actual rrd folder of ‘ubuntuhp’ is created NOT on ubuntus1 (server, 192.168.100.23) but on the poller itself ubuntuhp - thus like there is no distributed polling switched on. Yes, ubuntuhp tries to communicate via rrdcached but it creates everything locally…
Running discovery.php didn’t create all the rrd files (if it should?)

So, i’m stucked.
Is the configuration ok?
Should i use NFS or common file system between them? (i have understanding that it’s optional…)
What else have i missed?
How can i troubleshoot the issue to find the root cause?

Thank you a lot!

[abatapa]

Here is the output of poller run

https://pastebin.com/Bwq0Rj1C

line21: why does it create local directory?
Does it mean i need to have NFS?

[laf]

Your config is definitely wrong. For starters you look like you are running two entirely separate installs - they should share RRDCached (with NFS - or specify the rrdtool version https://docs.librenms.org/Extensions/RRDCached/ ), Memcached and MySQL to work.

Are you intending to split devices between pollers or do you want two pollers that can just poll all devices?

[abatapa]

Thank you, laf.
I want two pollers each with it’s own set of polled devices, but only one web gui (ubuntuS1 in my case).

rrdcached is working Ok on ubuntuS1 for it’s own devices vi TCP socket on port 42217.
The problem arises when other server (ubunthp) starts to connect to the rrdcached (192.168.100.23:42217)

let me clarify - “shared directory”, or NFS - is the only option that i must go in to?

Or there is a way to do everything without shared folder with NFS?

thank you a lot!

[laf]

You can use rrdcached with no shared FS, you just need to specify $config['rrdtool_version'] = '1.5.5';

Make sure they both use the same memcached and mysql server, without that it won’t work.

Your S1 config looks like it’s missing all the distributed polling config as well, you can take what you have for the hp install and re-use that, just change the variables necessary.

[abatapa]

thanks laf for helping me to move with it.
Yes, i’d like to have everything without NFS.

And that is because i’m asking

1. "you just need to specify $config[‘rrdtool_version’] = ‘1.5.5’;"
   where i should specify this? On both and server and a poller? Or only at the server? Or only at the poller?
   On my config it is specified on the server (as on manual for rrdcache) but it’s not on the poller (because it is not told there)

So - where exactly should i put rrdtool_version? Server? Poller? Both? Is it written somewhere?

2. regarding your:
   “Make sure they both use the same memcached and mysql server, without that it won’t work.”
   … as you can see from the config
   a) they DON’T use the same mysql server, because they both use their own librenms installation.

On the manual:
“Database Server
MySQL / MariaDB - At the moment these are the only database servers that are supported, work is being done to ensure MySQL Strict mode is also supported but this should be considered to be incomplete still and therefor disabled.
The pollers, web and API layers should all be able to access the database server directly.”

it is completely not clear, that they should connect to THE SAME MYSQL SERVER? In my case, if it’s required mysql server on ubuntus1 is ready to get the connections from ubuntuhp (all permissions granted).

So, do you mean, that on the poller (ubuntuhp) on the config file,
instead of: $config[‘db_host’] = ‘localhost’;
i should put:
$config[‘db_host’] = ‘192.168.100.23’; <- IP of the server (ubuntuS1, where memcached, mysql, rrdcache runs?)

Do i understand correctly?

b) they are yousing the same memcached server (ubuntus1, 192.168.100.23) - its displayed on the config above.

3. regarding S1 missing distributed config - just to confirm, that on ubuntuS1 i should also put row
   $config[‘distributed_poller’] = true;
   to make it “distributed ready”?

do i understand correctly? Or something else should be put?
(each row means something, each row can be the cause of it’s not working).

Thank you again!

[laf]

@abatapa

where i should specify this? On both and server and a poller? Or only at the server? Or only at the poller?

On my config it is specified on the server (as on manual for rrdcache) but it’s not on the poller (because it is not told there)

It needs to be set on all pollers in config.php

@abatapa

a) they DON’T use the same mysql server, because they both use their own librenms installation.

That’s not how this works, you HAVE to use the same MySQL, Memcached and rrdcached instances - you CANNOT have them separately.

@abatapa

it is completely not clear, that they should connect to THE SAME MYSQL SERVER? In my case, if it’s required mysql server on ubuntus1 is ready to get the connections from ubuntuhp (all permissions granted).

You’re more than welcome to edit the docs and improve them

@abatapa

So, do you mean, that on the poller (ubuntuhp) on the config file,

instead of: $config[‘db_host’] = ‘localhost’;

i should put:

$config[‘db_host’] = ‘192.168.100.23’; <- IP of the server (ubuntuS1, where memcached, mysql, rrdcache runs?)

Do i understand correctly?

Correct yes.

@abatapa

b) they are yousing the same memcached server (ubuntus1, 192.168.100.23) - its displayed on the config above.

regarding S1 missing distributed config - just to confirm, that on ubuntuS1 i should also put row

$config[‘distributed_poller’] = true;

to make it “distributed ready”?

You need to put all config for distributed polling, not just that. RRDCached, Memcached need to be set.

@abatapa

do i understand correctly? Or something else should be put?

(each row means something, each row can be the cause of it’s not working).

Sorry I’m not sure what you mean by that.

[abatapa]

THank you laf a lot. It works. it really works

So, here is the configuration:
(it is actual working config from both machines taken with ’ cat config.php | grep -e “#” ’ )

Server ubuntuS1 configuration (it’s ip is 192.168.100.23)
###########
$config[‘snmp’][‘community’] = array(“public”);

$config[‘db_host’] = ‘localhost’;
$config[‘db_port’] = ‘3306’;
$config[‘db_user’] = ‘librenms’;
$config[‘db_pass’] = ‘password’;
$config[‘db_name’] = ‘librenms’;
$config[‘db_socket’] = ‘’;
$config[‘user’] = ‘librenms’;

$config[‘memcached’][‘enable’] = TRUE;
$config[‘memcached’][‘host’] = “192.168.100.23”;
$config[‘memcached’][‘port’] = 11211;
$config[‘memcached’][‘ttl’] = 60;

$config[‘rrdcached’] = “192.168.100.23:42217”;
$config[‘rrd_dir’] = “/opt/librenms/rrd”;
###################

remote poller configuration
#####################
$config[‘db_host’] = ‘192.168.100.23’; #NB: !!! IP of ubuntuS1, so it connects to server
$config[‘db_port’] = ‘3306’;
$config[‘db_user’] = ‘librenms’;
$config[‘db_pass’] = ‘password’;
$config[‘db_name’] = ‘librenms’;

$config[‘user’] = ‘librenms’;
$config[‘rrdtool_version’] = ‘1.5.5’;

$config[‘distributed_poller’] = true;
$config[‘distributed_poller_name’] = “ubuntuhp”;
$config[‘distributed_poller_group’] = ‘2’; #NB: poller groups were created in GUI
$config[‘distributed_poller_memcached_host’] = ‘192.168.100.23’; #NB: connects to server
$config[‘distributed_poller_memcached_port’] = 11211;
$config[‘rrdcached’] = “192.168.100.23:42217”; #NB: connects to server

####################

Let me bring it up on two more machines from the scratch, and i’ll write smth in addition (there were more things which i fought )

thank you, laf!

[laf]

I hope you’re not using password for the mysql password

[jongalli]

Could you confirm whether ubuntuS1 configuration requires " $config[‘distributed_poller’] "… Or is this only specified for additional pollers such as ubuntuHP.

I am looking to setting up a remote poller - the only difference being that all poller groups would be polled by each server.

Thanks