Polling of devices very intermittent, sometime hours

Help
#1

Hello,

We have a LibreNMS server and 1 poller, but polling is very intermittent on all devices, can you help/advise please?

====================================
Component | Version
--------- | -------
LibreNMS  | 22.2.1-6-g703745d0e
DB Schema | 2022_02_03_164059_increase_auth_id_length (235)
PHP       | 7.4.3
Python    | 3.8.10
MySQL     | 10.3.32-MariaDB-0ubuntu0.20.04.1
RRDTool   | 1.7.2
SNMP      | 5.8
====================================

[OK]    Composer Version: 2.2.6
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[INFO]  Detected Dispatcher Service
[FAIL]  Missing PHP extension: memcached
	[FIX]: 
	Please install memcached
[WARN]  Your local git contains modified files, this could prevent automatic updates.
	[FIX]: 
	You can fix this with ./scripts/github-remove
	Modified Files:
	 includes/definitions/rutos.yaml
	 rrd/.gitignore

image
image1885×528 58.1 KB

This how all the CPU, Mem, network graphs look on all devices. This is one of the graphs for the LibreNMS server itself (local).

image
image875×343 12.7 KB

The poller is in Docker, I’m not sure how to validate it.

image
image2852×196 90.2 KB 

bash-5.1# cd /opt/librenms/

bash-5.1# ./validate.php

Do not run validate.php as root

What could be wrong?

#2

I think there are some sync problems between pollers. Are you using redis? memcached ? Is it configured correctly.
Second, i think number of workers are not enough. What happens if you increase the number ?

As best practice, for 1k+ devices, I would skip docker. Just me feelling here.

#3

Does that cluster health look bad then?

I thought I had memcache installed but says:

[FAIL]  Missing PHP extension: memcached
	[FIX]: 
	Please install memcached

How do I check and install?

I’m sure it ran better without the poller how can I check if they are syncing properly?

Noticed this too in the Global settings, should this be enabled?

image
image1097×309 9.65 KB

Also what config(s) can I share that might help you understand the setup?

#4

My experience with docker is almost none.
Can you validate the config from the web interface?

#5

I’m tempted to remove this docker poller then.

image
image1882×254 6.16 KB

I’ve restarted both VMs and it seems a little quicker, but it seems quicker with the poller off. However the graphs are still bad.

With the poller off and via CLI I see

[OK]    Composer Version: 2.2.6
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[INFO]  Detected Dispatcher Service
[FAIL]  Missing PHP extension: memcached
	[FIX]: 
	Please install memcached

I’m not sure where to start on this.

#6

I see this a lot too. I’ve turned that poller back on.

image

#7

For unknown reason your cluster is not capable to handle all polling in the configured timeframe (5 minutes standard).

If configuration is fine (not sure about this), take a look into this.

#8

I’ll go through this and get back to you.

To increase the workers where do I go for this?

I didn’t set all of this up so it’s hard and I’m under pressure to get it working so thanks for helping.

#9

If the configuration is not changed in the config file than it can be done in the web interface.

#10

I can post the config if you like maybe you will notice something odd?

#11

I ran the MySQL tuner, anything jump out?

https://pastebin.com/4sRe6SXe

How do I increase the workers and SQL connections?

#12

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.