Distributed poller set up

Help
1

Hi,

Is there a complete tutorial on distributed poller setup?
I followed the installation document http://docs.librenms.org/Installation/Installation-Ubuntu-1604-Nginx/ to setup the DB, web front, but would like to have one poller on each datacenter. How do I do that?

Any helps will be appreciated.

Thanks

2 Likes
2

Yep: http://docs.librenms.org/Extensions/Distributed-Poller/

3

Many Thanks @murrant

I am very new to this.
Is a poller generally means a librenms installation?
where should I install rrdcached and memcached? Can they both be installed on the web front end or the database server?

Thanks

4

Yes a poller is a normal install but you donā€™t need to run mysql, memcached and rrdcached locally. These should all be installed in one central place - where - is up to you.

5

image.png916Ɨ772 42 KB

Is my diagram of how iā€™m setting up my distributed LibreNMS setup, if that helps you visualize anything

6

Thanks All!
These are very helpful to me!

I was having some issues after I installed memcached and rrdcached. The graphs stopped drawing, the poller still polls, but nothings shows on the graph after I bounced nginx.

I put web front, rrdcached, memcached and a poller on one box.
I follow this doc to install and configure memcached http://docs.librenms.org/Extensions/Memcached/
this doc to install and configure rrdcache for ubuntu http://docs.librenms.org/Extensions/RRDCached/
Finally, I follow the document @murrant gave me to configure everything again.

Then after all of that, the graphing stopped working. When I validate my installation, I got this:
[FAIL] Cannot connect to rrdcached instance

Any clues?

Thanks

Harry

7

Run ./validate.php first of all.

Likely it is a file permission issue.

8

That is the output of the Validate.php
I put this configuration in the config.php file in /opt/librenms
Can someone kindly help me to verify?

# Memcached
$config['memcached']['enable'] = true;
$config['memcached']['host'] = "localhost";
$config['memcached']['port'] = 11211;

# RRDCACHED
$config['rrdcached'] = "unix:/var/run/rrdcached.sock";

#$config['distributed_poller'] = true;
#$config['rrdcached'] = "localhost:42217";

$config['distributed_poller_name'] = file_get_contents('/etc/hostname');
$config['distributed_poller_group'] = '0';
$config['distributed_poller_memcached_host'] = "localhost";
$config['distributed_poller_memcached_port'] = 11211;
$config['distributed_poller_host'] = "localhost";
$config['distributed_poller_port'] = 11211;
$config['distributed_poller'] = true;
$config['rrdcached'] = "localhost:42217";
$config['update'] = 0;

9

Remove the above.

Youā€™ve already posted the issue, we canā€™t connect to rrdcached so check itā€™s actually running on 42217 (netstat -anp|grep 42217).

10

Thanks @laf !
After commented out the section you mentioned above, I was still having issues. When I grep port 42217, its not there.
So I took out the line $config[ā€˜rrdcachedā€™] = 42217 , now the config looks like below and it works, at least no error on validation. But I wondering how will it work for other pollers if rrdcached port not open?

$config[ā€˜distributed_pollerā€™] = true;
$config[ā€˜rrdcachedā€™] = ā€œunix:/var/run/rrdcached.sockā€;
$config[ā€˜distributed_poller_nameā€™] = file_get_contents(ā€™/etc/hostnameā€™);
$config[ā€˜distributed_poller_groupā€™] = ā€˜0ā€™;
$config[ā€˜distributed_poller_memcached_hostā€™] = ā€œlocalhostā€;
$config[ā€˜distributed_poller_memcached_portā€™] = 11211;
$config[ā€˜distributed_poller_hostā€™] = ā€œlocalhostā€;
$config[ā€˜distributed_poller_portā€™] = 11211;
$config[ā€˜distributed_pollerā€™] = true;
$config[ā€˜updateā€™] = 0;

11

It wonā€™t but basically youā€™ve not configured rrdcached to start on port 42217 so you need to fix that first.

12

Got it! I uncommented NETWORK_OPTIONS="-L" and commented out SOCKFILE=/var/run/rrdcached.sock
in the config.php, I changed this line from $config[ā€˜rrdcachedā€™] = ā€œunix:/var/run/rrdcached.sockā€; to $config[ā€˜rrdcachedā€™] = ā€œ127.0.0.1:42217ā€;
Bounce rrdcached and it is now listening to port 42217.
Validated setup and no errors.

I assume that means its working?

Many Thanks for the help!

Continue to setup first poller.

1 Like
13

Fingers crossed it does mean itā€™s working :slight_smile:

14

It worked indeed!

So I spun up another poller and move the devices to that poller, it seems all the servers devices worked with that poller but not the networking devices. When I moved it back to the default pooler after a while, they all appear to be offline. I tried delete a device and re-add it, but wasnā€™t success, webserver timed out.

Any Clues?

Thanks

15

Most likely you arenā€™t allowing icmp / snmp from the new pollers IP.

16

yeah, I think thatā€™s most likely the case.

When I move them back to the default poller and they still not working. The default poller IP should be allowed on those devices. They were working before I moved them to the new poller and back.

Is there a way to delete a device and clean all the memcached and db history about that device and treat it as new?

Thanks

Harry

17

Data like that isnā€™t stored in memcache. Removing a device should clear all itā€™s data out.

If you run ./poller.php -h HOSTNAME -d -m os -r -f it will give you some debug on whatā€™s wrong.

18

I am very confused on setting up the poller.
My current config for a poller.(see below)
172.16.37.209 is where the master poller/memcached/rrdcached located and they use 11211 and 42217 make sense because this host need to get to those services via these ports.

The lines that threw me off was $config['distributed_poller_host'] and $config['distributed_poller_port'].
I put this pollerā€™s IP on this line and put 11211 in the port section, but I felt its not right. What is port 11211 is used for in this case? Does this connect back to the memcached server?
If I put 172.16.37.209 there, then it still not making sense since its asking for the distributed_poller_host.

Any helps will be very much appreciated!

$config['distributed_poller_name'] = file_get_contents('/etc/hostname');
$config['distributed_poller_group'] = '1';
$config['distributed_poller_memcached_host'] = "172.16.37.209";
$config['distributed_poller_memcached_port'] = 11211;
$config['distributed_poller_host'] = "172.30.16.102";
$config['distributed_poller_port'] = 11211;
$config['distributed_poller'] = true;
$config['rrdcached'] = "172.16.37.209:42217";
$config['update'] = 0;

19

Those are wrong. Ignore that the docs need updated. Just remove those two lines.

20

Thanks all!
Finally got my distributed pollers setup.
All thanks to you guys!

Cheers