I’m managing a decent sized LibreNMS deployment (100+ monitored devices) and I noticed on Friday we started seeing a huge performance regression in LibreNMS that caused pages to take forever to load and rrdcached to hit 300% cpu utilization consistently each time someone browsed between devices.
Today we came back to the server still being slow so I made the executive decision to rollback the server to the latest “release” at 21.11.0. The version we were seeing issues was 21.11.0-66-g528d8e55b, this would also include any daily updates since Friday, December 3rd.
Since the rollback, performance has returned back to normal and the site is loading quickly again. This issue was not isolated to just one server, we manage multiple smaller instances and we were seeing this issue replicated across the board. Likewise, once we rolled back to 21.11.0 everything was good again.
Has anyone else noticed their instance loading slower than usual?
I don’t think there’s any general change in the code that would have that effect on performance. We have close to 4,000 devices and I have not seen any performance drain. Go through the change log and see if there’s any specific piece of code that has to do with your implentation that could cause this.
I have the same problem and it still exists in version 21.12.1. The All devices page takes a long time to load. How can I roll back to version 21.11.0?
Probably not very helpful but we are running version 21.12.1 with 343 monitored devices and I’m not seeing any slowdowns relative to any previous versions. All pages are snappy to load, and the “server” only runs on a repurposed desktop class PC not server class hardware.
It looks to be something to do with instances that are configured to use RRDCached to generate graphs.
I experimented with turning this feature off for LibreNMS and falling back to it’s original RRDTool graphing and saw a huge improvement in performance. Not sure what could have caused it but hopefully this information can help someone track it down.
You can rollback to version 21.11.0 by going to your LibreNMS server and running the following commands as the librenms user (from within the librenms server directory /opt/librenms/)
git checkout tags/21.11.0
But before you do that, if you have LibreNMS configured to use rrdcached, I suggest just commenting out that option in the LibreNMS config file and try loading your pages again. In the environments I was seeing this issue crop up, this was what brought performance back while being able to stay up to date.
I’m also running into a huge performance issue, sometimes my pages won’t even load and I get a Gateway Timeout. My CPU and memory are maxed out at 97%. I checked the rrdcached setting in my config file and I have it commented out. I guess I will try rolling back to 21.11.0
Rolling back to 21.11.0 didn’t help. I can’t even pull up the web interface anymore. I see another reload in my future, does anyone know if there is a post about backing up and reloading a database on a new instance?