LibreNMS - v24.4.1- Long polling time & High CPU usage

Saguu · 2 May 2024 16:03

Hello,

I encounter multiple problems with LibreNMS, which seem to be related to polling time:

The output of ./validate.php :

bash-5.1$ ./validate.php

Component	Version
LibreNMS	24.4.1-27-gfe2f8a6c8 (2024-05-01T20:15:27+01:00)
DB Schema	2024_04_22_161711_custom_maps_add_group (292)
PHP	8.1.27
Python	3.9.18
Database	MariaDB 10.5.22-MariaDB
RRDTool	1.7.2
SNMP	5.9.1
===========================================

[OK] Composer Version: 2.7.4
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK] Database and column collations are correct
[OK] Database schema correct
[OK] MySQL and PHP time match
[FAIL] No active polling method detected
[OK] Dispatcher Service not detected
[OK] Locks are functional
[FAIL] No active python wrapper pollers found
[OK] Redis is unavailable
[OK] rrdtool version ok
[OK] Connected to rrdcached
[FAIL] We have found some files that are owned by a different user than ‘librenms’, this will stop you updating automatically and / or rrd files being updated causing graphs to fail.
[FIX]:
sudo chown -R librenms:librenms /opt/librenms
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
Files:
/opt/librenms/rrd/smokeping/__sortercache/data.lnmsFPing-1.storable
/opt/librenms/rrd/smokeping/__sortercache/data.lnmsFPing-0.storable
bash-5.1$

I have a VM with 8 CPU/32 Gb RAM and 256 Gb SSD, with graph losses and high polling times.

Also, I see this at regular intervals:

[FAIL] No active polling method detected
[FAIL] No active python wrapper pollers found

Could you help me solve this?

Thanks

murrant · 6 May 2024 01:31

Likely some sort of configuration issue. I suggest reviewing the install docs and checking that your install matches.

Saguu · 6 May 2024 07:24

Thanks for your help.

I have already gone through the installation procedure and tuned the DB using MySQLTuner, the performance has improved but I still have high CPU usage and an average polling time of 60 seconds per device.
I have fewer holes in the graphs but it is still present.

Any other suggestions ?

murrant · 7 May 2024 01:30

Did you review which devices are taking the longest and see if you can figure out why?

Saguu · 7 May 2024 10:01

Thanks for your help.

I just carried out a manual poll, but I’m not sure how to use the information below?

Thanks

bash-5.1$ lnms device:poll 113
Starting polling run:

Hostname: X.X.X.X (cisco)
ID: 113
OS: iosxe
IP: X.X.X.X

Load poller module core

SNMP: [3/1.53s] MySQL: [4/0.26s] RRD: [4/0.00s]
Runtime for poller module ‘core’: 1.1407 seconds with 121512 bytes

Unload poller module core

Load poller module os

SNMP: [0/0.00s] MySQL: [1/0.01s] RRD: [2/0.00s]
Runtime for poller module ‘os’: 0.0168 seconds with 189656 bytes

Unload poller module os

Load poller module availability

SNMP: [0/0.00s] MySQL: [13/0.43s] RRD: [10/0.01s]
Runtime for poller module ‘availability’: 0.0748 seconds with 74528 bytes

Unload poller module availability

Load poller module sensors

SNMP: [9/5.38s] MySQL: [29/0.77s] RRD: [126/0.12s]
Runtime for poller module ‘sensors’: 5.6471 seconds with 176360 bytes

Unload poller module sensors

Load poller module processors

SNMP: [1/0.61s] MySQL: [1/0.04s] RRD: [20/0.01s]
Runtime for poller module ‘processors’: 0.6410 seconds with 72736 bytes

Unload poller module processors

Load poller module mempools

SNMP: [1/0.57s] MySQL: [1/0.01s] RRD: [6/0.00s]
Runtime for poller module ‘mempools’: 0.5814 seconds with 86368 bytes

Unload poller module mempools

Load poller module storage

SNMP: [1/0.56s] MySQL: [2/0.02s] RRD: [4/0.00s]
Runtime for poller module ‘storage’: 0.5714 seconds with 9040 bytes

Unload poller module storage

Load poller module netstats

SNMP: [11/5.21s] MySQL: [0/0.00s] RRD: [12/0.01s]
Runtime for poller module ‘netstats’: 5.2296 seconds with 48936 bytes

Unload poller module netstats

Load poller module hr-mib

SNMP: [1/0.47s] MySQL: [0/0.00s] RRD: [2/0.00s]
Runtime for poller module ‘hr-mib’: 0.4683 seconds with 2384 bytes

Unload poller module hr-mib

Load poller module ucd-mib

SNMP: [2/0.94s] MySQL: [0/0.00s] RRD: [2/0.00s]
Runtime for poller module ‘ucd-mib’: 0.9446 seconds with 3712 bytes

Unload poller module ucd-mib

Load poller module ipSystemStats

SNMP: [1/1.40s] MySQL: [0/0.00s] RRD: [6/0.00s]
Runtime for poller module ‘ipSystemStats’: 1.4050 seconds with 5064 bytes

Unload poller module ipSystemStats

Load poller module ports

SNMP: [26/18.34s] MySQL: [30/0.38s] RRD: [50/0.07s]
Runtime for poller module ‘ports’: 18.6504 seconds with 347248 bytes

Unload poller module ports

Load poller module customoid

SNMP: [0/0.00s] MySQL: [1/0.01s] RRD: [2/0.00s]
Runtime for poller module ‘customoid’: 0.0105 seconds with 1032 bytes

Unload poller module customoid

Load poller module bgp-peers

SNMP: [16/7.08s] MySQL: [17/0.20s] RRD: [22/0.02s]
Runtime for poller module ‘bgp-peers’: 7.1345 seconds with 243256 bytes

Unload poller module bgp-peers

Load poller module ucd-diskio

SNMP: [0/0.00s] MySQL: [1/0.07s] RRD: [2/0.00s]
Runtime for poller module ‘ucd-diskio’: 0.0081 seconds with 640 bytes

Unload poller module ucd-diskio

Load poller module cisco-ipsec-flow-monitor

SNMP: [1/2.81s] MySQL: [0/0.00s] RRD: [2/0.00s]
Runtime for poller module ‘cisco-ipsec-flow-monitor’: 2.8089 seconds with 4440 bytes

Unload poller module cisco-ipsec-flow-monitor

Load poller module cisco-remote-access-monitor

SNMP: [1/0.43s] MySQL: [0/0.00s] RRD: [2/0.00s]
Runtime for poller module ‘cisco-remote-access-monitor’: 0.4337 seconds with 2576 bytes

Unload poller module cisco-remote-access-monitor

Load poller module cisco-cef

SNMP: [4/5.11s] MySQL: [7/0.06s] RRD: [8/0.00s]
Runtime for poller module ‘cisco-cef’: 5.1263 seconds with 161160 bytes

Unload poller module cisco-cef

Load poller module slas

SNMP: [0/0.00s] MySQL: [1/0.01s] RRD: [6/0.00s]
Runtime for poller module ‘slas’: 2.6899 seconds with 49800 bytes

Unload poller module slas

Load poller module cisco-mac-accounting

SNMP: [0/0.00s] MySQL: [1/0.01s] RRD: [2/0.00s]
Runtime for poller module ‘cisco-mac-accounting’: 0.0030 seconds with 4200 bytes

Unload poller module cisco-mac-accounting

Load poller module cipsec-tunnels

SNMP: [1/0.73s] MySQL: [1/0.01s] RRD: [2/0.00s]
Runtime for poller module ‘cipsec-tunnels’: 0.7387 seconds with 1315240 bytes

Unload poller module cipsec-tunnels

Load poller module cisco-voice

SNMP: [5/3.29s] MySQL: [0/0.00s] RRD: [4/0.00s]
Runtime for poller module ‘cisco-voice’: 3.2910 seconds with 8992 bytes

Unload poller module cisco-voice

Load poller module cisco-cbqos

SNMP: [0/0.00s] MySQL: [1/0.01s] RRD: [2/0.00s]
Runtime for poller module ‘cisco-cbqos’: 0.0037 seconds with 82632 bytes

Unload poller module cisco-cbqos

Load poller module cisco-otv

SNMP: [0/0.00s] MySQL: [1/0.01s] RRD: [2/0.00s]
Runtime for poller module ‘cisco-otv’: 0.0033 seconds with 12056 bytes

Unload poller module cisco-otv

Load poller module cisco-qfp

SNMP: [2/1.11s] MySQL: [10/0.14s] RRD: [6/0.00s]
Runtime for poller module ‘cisco-qfp’: 1.1401 seconds with 406944 bytes

Unload poller module cisco-qfp

Load poller module cisco-vpdn

SNMP: [1/0.49s] MySQL: [0/0.00s] RRD: [2/0.00s]
Runtime for poller module ‘cisco-vpdn’: 0.4913 seconds with 2760 bytes

Unload poller module cisco-vpdn

Load poller module entity-physical

SNMP: [0/0.00s] MySQL: [0/0.00s] RRD: [2/0.00s]
Runtime for poller module ‘entity-physical’: 0.0010 seconds with 2112 bytes

Unload poller module entity-physical

Load poller module applications

SNMP: [0/0.00s] MySQL: [1/0.01s] RRD: [2/0.00s]
Runtime for poller module ‘applications’: 0.0045 seconds with 139376 bytes

Unload poller module applications

Load poller module stp

SNMP: [0/0.00s] MySQL: [2/0.01s] RRD: [2/0.00s]
Runtime for poller module ‘stp’: 0.0035 seconds with 78984 bytes

Unload poller module stp

Load poller module ntp

SNMP: [1/0.87s] MySQL: [4/0.03s] RRD: [6/0.00s]
Runtime for poller module ‘ntp’: 0.8782 seconds with 31264 bytes

Unload poller module ntp

Enabled graphs (18): uptime poller_modules_perf availability netstat_icmp netstat_icmp_info netstat_ip netstat_ip_frag netstat_snmp netstat_snmp_pkt netstat_udp netstat_tcp ipsystemstats_ipv4 ipsystemstats_ipv4_frag ipsystemstats_ipv6 ipsystemstats_ipv6_frag poller_perf ping_perf cisco-voice-ip

Start Alerts

Rule #39 (IPSec tunnels down):
Status: NOCHG
Rule #52 (Interface Errors Rate greater than 100):
Status: NOCHG
Rule #65 (Device Down - No ICMP response):
Status: NOCHG
Rule #66 (BGP Session down):
Status: NOCHG
Rule #70 (Device Down - SNMP unreachable):
Status: NOCHG

End Alerts (0.0277s)

Start Device Groups

End Device Groups (0.0136s)

Polled XXXX (113) in 61.874 seconds <<<

SNMP [88/56.93s]: Snmpget[46/22.95s] Snmpwalk[42/33.98s]
SQL [151/2.80s]: Select[77/1.80s] Update[70/0.95s] Delete[1/0.01s] Insert[3/0.03s]
RRD [324/0.28s]: Other[162/0.18s] Update[162/0.10s]

Blockquote

Peter_Persson · 8 May 2024 08:34

If you look at /poller/log via webgui, you see what devices take longer time.
Then you can look at that device on what module thats to long time.
Some vendors/versions have poor snmp implementations which can create this issue.

zertap · 13 May 2024 03:34

I am also experiencing increased CPU usage & polling times after upgrading to v24.4.1 (from 23.x).

2 week graph from poller performance. Update date is clearly visible.

pkill · 15 May 2024 16:44

We’re seeing the same thing since our system updated on May 6.

murrant · 15 June 2024 02:10

One thing that can cause this is code that has errors in it and having error reporting enabled.

Two fixes:

report and fix the errors. (outside error reporting)
disable error reporting