New Poller Service [BETA] : are there some better docs available?

Help
1

The pre-amble for the New Poller Service [BETA] doc says "If you are currently using the old poller service, it’s strongly recommended that you migrate away… ".

I note that the old poller service is still part of the standard install, as the above doc also says “Once the poller service is installed, the cron scripts used by LibreNMS are no longer required and must be removed.”

Are there some better docs, as the above doc is lacking in a number of key areas:

  • when does this new poller service come out of beta & when is it recommended to replace the current service? This info should be nearer the top of the docs stack & incorporated into any additional relevant docs (e.g. the distributed poller doc).

  • “Make sure you uninstall the old poller service before deploying the new one.” : no instructions on how to do this.

  • " Redis (distributed polling only)" : this appears to supersede the current distributed pollers doc, but lacks sufficient info to install and configure redis.

  • “It’s recommeded that you do not share the redis database with any other system” : what does “system” mean in this context?

I’m looking into distributed pollers now, so am keen to follow the recommended solution.

Cheers.

1 Like
2

Any news on this guys?

I’m busy adding systems to my LibreNMS instance, which means the poller’s getting more and more hammered. If this beta way is the right way, then can it be made production & receive some proper docs?

3

I moved to the service a couple of months ago and it is a good improvement and stable.

Stuff that is important but may have skipped
BACKUP FIRST!
Do all of this as the LibreNMS user
Read all of this and linked docs before starting

Link to official Docs: Redirecting...

My Notes will be for Ubuntu if you need something else I can help find the commands for your system.

The first thing you will need to do, disable all current LibreNMS cron jobs.
Here is how you may have set it up…
https://docs.librenms.org/Installation/Installation-Ubuntu-1404-Apache/#create-cronjob

33   */6  * * *   librenms    /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1
*/5  *    * * *   librenms    /opt/librenms/discovery.php -h new >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/cronic /opt/librenms/poller-wrapper.py 16
*    *    * * *   librenms    /opt/librenms/alerts.php >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/poll-billing.php >> /dev/null 2>&1
01   *    * * *   librenms    /opt/librenms/billing-calculate.php >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/check-services.php >> /dev/null 2>&1

Comment them out if you like until you are sure everything works.

Next: Install and or update Python

It will likely be something like this

apt update && apt upgrade
apt install python3
apt install python3-pip

Next: Python modules
This can go well or take you down a long path. This command worked well on 18.04 but I had one system on 16.04 and I remember some errors and I have to hunt down some extra stuff to install.

pip3 install -r requirements.txt

Next: Redis
I found this to be easy but I choose not to secure with a password since I am on a corp network. I did however like most things isolated it via firewall rules, which I will include. Follow these institutions and you should be fine.

Since I am on Ubuntu I use UFW and only allow the NMS poller servers to talk into Redis.

ufw allow from 1.2.3.4 to any port 6379

Note: This would be executed on the Redis server and the IP 1.2.3.4 would be the NMS poller server.

Next: MySQL
I may have glossed over this so let’s hit it now quickly. I am currently polling about 650 devices and over 27k ports. Take a look at Scaling LibreNMS Scaling LibreNMS - LibreNMS Docs and consider dedicating servers for individual components. The short version is that Pollers are just normal LibreNMS installs that point to a shared SQL, RRD, Redis, and Memcache. (more on this memcached thing later.

Next: Distributed Polling Configuration
This is straight forward and from the existing docs.
Note: Make sure you did everything under LibreNMS user account

Once you have your Redis database set up, configure it in the .env file on each node.

APP_KEY=xxxxxxxxxxxxxxxxxxxx

DB_HOST=1.2.3.9
DB_DATABASE=librenms
DB_USERNAME=librenms
DB_PASSWORD=xxxxxxxxxxxxxx

APP_URL=/
NODE_ID=xxxxxxxxxx
LIBRENMS_USER=librenms

REDIS_HOST=1.2.3.8
REDIS_DB=0
REDIS_PASSWORD=
REDIS_PORT=6379

Next: Basic Configuration
This step is straight forward, nothing too tricky.
https://docs.librenms.org/Extensions/Poller-Service/#basic-configuration
After you are set up you can tweak the number of workers until you max out resources.

Next: Service Installation
Again a straight forward step.
https://docs.librenms.org/Extensions/Poller-Service/#service-installation

Final
At this point, you should be done. Head over to https://librenms.local/pollers/tab=pollers/ and see your new workers. Now this goes to your question is the new poller service ready and when will it be out of beta. I do not think you need memcached as it appears the new service is using Redis. However, I left mine in place since the validate script will complain if it is not there. So that is an example of why it says Beta, maybe memcached is not needed and at some point the validate script will be updated.

I hope this helps and good luck.

2 Likes
4

Great answer :smiley:
I’ll give it a go.

The odd thing now is that validate says the poller isn’t finishing in 5mins (hence distributed & new poller interest) , but manually running the poller with 32 threads results in a 1 min ish runtime.

5

Take a look at config.php and tweak the settings to get where you like. There was an old post somewhere that recommended two poller workers per CPU core as a starting point. I run 16 cores with 32 workers and CPU spikes to about 60% for less than 30 seconds every polling cycle. I have gone higher but I have everything on separate VMs with Isolated VLANs carrying the SQL, RRD, and cache traffic. So experiment and see what you can get. Just remember that it’s a service now so (systemctl restart librenms).

$config[‘service_poller_workers’] = 24; # Processes spawned for polling
$config[‘service_services_workers’] = 8; # Processes spawned for service polling
$config[‘service_discovery_workers’] = 16; # Processes spawned for discovery

Consider disabling poller modules to save time? If you are not using or interested in BGP or OSPF then why ask your devices to tell you about them?

$config[‘poller_modules’][‘bgp-peers’] = false;
$config[‘poller_modules’][‘ospf’] = false;

Here are the defaults for the poller modules, see if you can cut some unneeded modules globally and modify the config.php.

Take a look at Pollers > History from the menu and see who is slow. From that menu click on the device and you will see the poller performance for that device. Maybe that system is very old and it just has issues answering questions in a timely fashion. Again here on a per-device basis you can disable/enable poller modules.

Take a look at Performance optimizations for some good tips. I have some slow link offices and " fping tuning" the timeout was helpful.

Note: “Performance optimisations” spelling error; I only caught it due to copy-paste-spellcheck.