Poller stops intermittently

I’m experiencing something that looks like halts in the polling at certain intervals. In the graphs it looks like at 00:00 the poller halts for a few minutes leaving a vertical gap in some of the graphs.

  • This happens very day
  • A large portion of the devices are affected
  • Not the same devices every time
  • The devices were not rebooted
  • They were not flagged as down thus creating an entry in the notification log
  • I have no reason to believe that underlaying network errors occur. Devices on the same net are affected, with some showing a gap in their graph, while neighbouring devices does not
  • The devices are servers, networking equipment, UPS’s, PDU’s, environment sensors and more. They are affected equally
  • There are no cron jobs running at 00:00 on the librenms box.

./discovery.php -h HOSTNAME -d → Untitled - LibreNMS
./poller.php -h HOSTNAME -r -f -d → Untitled - LibreNMS

librenms@librenms:~$ ./validate.php

Component Version
LibreNMS 21.4.0-30-g317705254
DB Schema 2021_04_08_151101_add_foreign_keys_to_port_group_port_table (208)
PHP 7.4.3
Python 3.8.5
MySQL 10.3.25-MariaDB-0ubuntu0.20.04.1
RRDTool 1.7.2
SNMP NET-SNMP 5.8
====================================

[OK] Composer Version: 2.0.12
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct

Example:
image

There are more examples, but tue to conspicuous names embedded in the images i start with this.

  • How many devices you have?
  • Are you using distributed poller ?
  • How is the CPU during that time ?
  • Mysql is in the same server with poller1 ?

Hi

  • ~70 devices
  • no
  • no spikes
  • yes

I think is related to a slow performance of your system where librenms is installed.
If you are not using distributed poller, you have a cron with daily.sh that is also clearing some events, logs and others. That might increase the load of mysql and poller and you see these gaps.

It happened to me long time ago.

https://docs.librenms.org/Support/FAQ/#why-do-i-see-gaps-in-my-graphs

It might be, but as far as i can see, the CPU is lurking around the 0% line. And it has 8 cores / 16gb Xeon v2.

I include som poller stats - does this look alarming? Should it be distributed?

As new users are only allowed one image pr post, here is the breakdown on pollers.

This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.