./daily.sh - Cleaning UP DB is suddenly taking a long time

Help
1

I have been running LibrenNMS for about a year without any issues.

I have recently noticed that when running the daily.sh script the update hangs for a long time on Cleaning UP DB

This used to be a very quick process, but now takes hours and hours.

I am using LibreNMS for my home network in which I have about 40 devices. I also monitor my parents and other relatives, which adds another 7 devices

Any help giving me some direction on where to start checking would be great…

  • The hardware:
  • LIbreNMS running as a VM in Proxmox on a R620 with dual E5-2680v2 128Gb or memory - raid 5 SSD
  • VM has plenty of resources and the storage system is decently fast
  • Network is 10gb between all the servers and VM’s and 1gb out to the none VM clients
2

./daily.sh shouldn’t be run manually.

Check and see what you have set for your clean up options.

3

As Kevin said, daily.sh should be running by cron everyday.

If you actually have that way, you should check for a device filling your syslog table.

4

I did find in syslog where the daily.sh job runs

However, I am still not sure why it is taking over 3 hours to complete

  • running it manually it sits @ Cleaning UP DB for the 90% of the time

At that time the load an cpu utilization of the VM are almost maxed out.

  • VM has 12gb or memory and 6 virtual CPU’s (I increased this from 6gb of memory and 4 vCPU a little while back to see if it would help anything… it has not)
5

Hi,

Just wondering if you got to the bottom of this, as I’m having the same issue: Stuck at “Cleaning up DB” for hours on end, with the associated php process taking CPU and swallowing memory as it goes.

EDIT: I’ve worked this out: It was stuck on the “recalculate_device_dependencies” section of the daily script. Turns out it doesn’t help to have devices that point to each other as a dependency … my daily update was stuck in an infinite loop, endlessly SELECTing and UPDATEing the max_depth column in the devices table. After fixing the device dependencies, the script runs fine.

6

Glad you found the issue.

In my case I looked a “validate config” one day and saw that it was telling me I had sql database issues.

I ran the commands it told me and since than everything seems to run correctly again…

I will mark this thread as resolved

7

Hi Vshaulsk,

I hava same issue. can you share your resolution to me? many thanks.

8

@Xin_Guo ./validate.php your install.

9

I was having this problem to, but it’s either a bug or a feature request.

We have a routing OSPF backbone and each core router has multiple connections to adjacent routers, so in theory a router is dependent on other routers being up and visa versa.

So the network engineers had setup LibreNMS with the routers being dependent on each other and basically got us into the recalculate loop.