Seems like something happened recently that is causing gaps in the data/graphs. I’ve had LibreNMS running for a while now with nothing like this happening until recently, like in the last day or two.
I’ve restarted my servers/router/etc and it persists. See the attached screen cap. the gaps are exactly 10 minutes every time.
Maybe the recent update has an issue?
This means that the poller is not completing in 5minutes consistently. Check your poller logs and performance graph. It could be that you added enough devices to exceed your hardware or a firmware update reduced the performance of SNMP on the device.
http://docs.librenms.org/Support/Performance
If you suspect it is an update, disable updates and roll back. Report an issue if you can pinpoint a commit.
I’ll look through all this and see what I can find, but literally, the only change made was the recent LibreNMS auto update. There have been no devices added for over a month. These gaps are appearing on graphs are from multiple devices, a pfsense firewall, a Synology NAS, a wifi access point etc.
Hard to believe that they all instantly, over night, started having issues. Only common factor is the recent LibreNMS update.
LibreNMS is running in a VirtualBox VM if that makes any difference.
I had something like this a few weeks ago. Tracked it down to a concurrent connection limit in my Firewall. When this as fix, the graphs were fine again
The code is always changing and we are always adding more metrics. If your install is teetering on the edge of having issues then an update can push it over.
Well, it’s the downloaded default VirtualBox Unbuntu VM…
That has nothing to do with it.
So I rolled back to the previous version, “6ff3ae8 - Thu Jan 12 2017 21:47:13 GMT-0800” (I’ve turned off the auto updates) and as you can see the data gaps have gone away. I changed nothing else on my network, the only difference is the update in to LibreNMS itself.
What version were you on?
Have you checked your poller graphs to show total poll times for the devices and which modules? Post these graphs
I was on version: “e90875b - Tue Jan 31 2017 00:09:56 GMT-0800”
Here are graphs from two of the devices:
That seems to have done it. Here’s graphs from the “e90875b - Tue Jan 31 2017 00:09:56 GMT-0800” version, gaps have gone away. You can see in the beginning of the charts when I edited the config file.
Do you mind explaining what was going on and how that geoloc fixed it?
So with that option enabled we do a lookup to the google api to translate addresses to geo coords. I’m guessing your install has no external access or at least has an issue connecting to the google api.
We have a pr in the queue which adds timeouts to these calls so it would be less impactful for the future.
There should be no reason why my install can’t reach the Google API. What are the IP address(s) LibreNMS is trying to reach?
Only thing i can think of is that I force Google Safe Search at the router. Or that Snort/Squid/Squidguard is blocking it for some reason.
Can you provide the IP address(s)?
Thanks, this worked for me too.
Where do you find those modules in CLI? I went to /opt/librenms/config.php and there are no modules in there to disable.
I had false values in some sensors (power supply on Odroid HC2) and up/down status since about a week (no LibreNMS update though) and that fixed it. Maybe Google changed something in their API?
Anyway, I’d never have guessed this was a geoloc issue… Thanks.
I have the same issue, but it is only in one trunk between a FW and an agregator.
I have a lot of another devices but this problem is only here.
Your probably running into this problem since its zeroing out >100G
thanks a lot for the answer, this was the problem.
Cheers
