High Load - Sunday Midnight

Hey everyone,

I’m trying to locate the cause of high system load occurring every week starting at 00:00 on Sunday and ending around 04:00 that same morning. The issue doesn’t occur on any other day - I can’t see any cron entries that run jobs specifically at midnight on Sunday.

The issue is causing delays in polling which triggers false alerts to be sent to the on-call engineer.
I suspect it might be something to do with the daily script?

I’m still going through the logs on our end but was curious if anyone else notices higher load at the same times?

$ ./validate.php

Component Version
LibreNMS 1.54-29-gcec10c1d3
DB Schema 2019_07_09_150217_update_users_widgets_settings (140)
PHP 7.2.19-0ubuntu0.18.04.1
MySQL 10.1.40-MariaDB-0ubuntu0.18.04.1
RRDTool 1.7.0
SNMP NET-SNMP 5.7.3
====================================

[OK] Composer Version: 1.9.0
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[WARN] Your local git contains modified files, this could prevent automatic updates.
[FIX]:
You can fix this with ./scripts/github-remove
Modified Files:
includes/definitions/discovery/ceraos.yaml
includes/definitions/discovery/eltek-webpower.yaml

LibreNMS is yust monitoring…

What mankes you think its not aan other process? (like weekly backup or so)

I dont have those spikes on my librenms

Don’t see anything like that on my nms in last month’s graph, the cpu usage is without spikes across the whole graph (monitoring 22 devices).

Can’t you ssh into the machine as it’s happening and monitor the cpu usage with htop or such ? I realise it’s another week before you could do that, but that’s all I got :wink:

Thanks for the reply, I’m aware it’s just monitoring haha.

Incremental jobs run between 10PM-11PM and on Saturday at 11PM a synth full task was running which completed at 4:54AM. This load is on our backup server (consolidating incremental images into a synthetic full - this doesn’t put any load on our LibreNMS Server).

The backup job on that night started at 11:10:08PM and finished at 11:17:10PM which involved a VM snapshot, the backup and snapshot removal. All of which finished before the midnight load on the LibreNMS box.

Thanks for the response, I might have to check if there is processor monitoring. I have 3 LibreNMS machines in production and this is the only one having the issue.