Poller Not Working Properly -- Running Slow

Help
1

I’m polling 3 devces:

Last week all of a sudden librenms / poller specifically just starting having issues.

Large poll times (maybe infinite) and huge gaps in graphs.

Tried basic steps, a reboot of the box / services

Polling all devices from cli and that itself is taking a long time, usual poll time was around ~8s

Not sure what happened, also removed the poller from the webui and was going to re-add but not sure how to do that.

From the looks of it my install appears borked.

validate.php
warning: unable to access '/home/a/.config/git/config': Permission denied
warning: unable to access '/home/a/.config/git/config': Permission denied
warning: unable to access '/home/a/.config/git/config': Permission denied
warning: unable to access '/home/a/.config/git/config': Permission denied
warning: unable to access '/home/a/.config/git/config': Permission denied
warning: unable to access '/home/a/.config/git/config': Permission denied
====================================
Component | Version
--------- | -------
LibreNMS  | 1.61-18-gc4eb412e2
DB Schema | 2020_02_10_223323_create_alert_location_map_table (159)
PHP       | 7.2.24-0ubuntu0.18.04.3
MySQL     | 10.1.44-MariaDB-0ubuntu0.18.04.1
RRDTool   | 1.7.0
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.9.3
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[FAIL]  The poller has never run or you are not using poller-wrapper.py, check the cron job.
[WARN]  Some devices have not been polled in the last 5 minutes. You may have performance issues.
	[FIX]:
	Check your poll log and see: http://docs.librenms.org/Support/Performance/
	Devices:
	 131.x.x.x
	 131.x.x.x
	 localhost
[FAIL]  Some devices have not completed their polling run in 5 minutes, this will create gaps in data.
	[FIX]:
	Check your poll log and see: http://docs.librenms.org/Support/Performance/
	Devices:
	 131.x.x.x
	 131.x.x.x
	 localhost
warning: unable to access '/home/a/.config/git/attributes': Permission denied
[WARN]  Your local git contains modified files, this could prevent automatic updates.
	[FIX]:
	You can fix this with ./scripts/github-remove
	Modified Files:
	 bootstrap/cache/.gitignore
	 logs/.gitignore
	 rrd/.gitignore
	 storage/app/.gitignore
	 storage/app/public/.gitignore
	 storage/debugbar/.gitignore
	 storage/framework/cache/.gitignore
	 storage/framework/cache/data/.gitignore
	 storage/framework/sessions/.gitignore
	 storage/framework/testing/.gitignore
	 storage/framework/views/.gitignore
	 storage/logs/.gitignore
[FAIL]  Some folders have incorrect file permissions, this may cause issues.
	[FIX]:
	sudo chown -R librenms:librenms /opt/librenms
	sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
	sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
	Files:
	 /opt/librenms/bootstrap/cache/services.php
2

Hi,

If you manually run ./poller.php -d -h anyhost you will get how many time the poller used in snmp, rrd and mysql.

Start from there to debug why is taking so many time.

3

Really odd, but removed my prometheus pushgateway config and things look better.

Running that command, it halted at the prometheus piece so anyway obvious change, will monitor.

4

pushgateway functionality is important since most people outside of our team use prometheus and grafana… has there been any additional work on that piece or still Alpha/Beta?

5

Afaik, still alpha/beta but I believe something is going on and probably will see the light very soon for testing.

I use influxdb without any issue, only when influx is down or working really low, 100% due to disk IO.

So, make sure that prometheus is working fine and you dont have connectivity issues betwen it and your librenms

6

Exactly what I’m verifying with the team that manages the prometheus infrastructure.

Thank you again!