The device having an issue.
It’s more than 10 systems, and just to clarify, event log… from the GuestOS or from libreNMS or? Hence asking what perspective, and if from GuestOS which log specifically?
I also want to add some details today that I’ve observed…
In this particular environment it’s a Proxmox VE Cluster. One node uses a consumer CPU AMD FX-8320 and the other is a Dell R720 with 2x E5-2667 v2’s for CPUs.
The FX-8320 node, all the VMs on it keep their CPUs after rebooting (in libreNMS)
The R720 node, every VM (except one) lose their CPUs after rebooting (in libreNMS). I can’t really see a pattern or why the single VM is keeping its CPUs while the others are losing them.
This is based on both Proxmox VE nodes rebooting for updates today, so this is current information. And each VM is fully updated too, with Ubuntu 20.04/22.04/24.04 flavours, and a few FreeBSD examples. So uhhh yeah not so easy to see a pattern just yet.
Eventlog in LibreNMS.
I’m not exactly seeing an obvious way to download the eventlog on the eventlog section of libreNMS… also… how can I get this to you securely since it contains sensitive info? (and yes I did try to google-fu downloading the eventlog before posting this comment).
If you want a raw and radected-ish snippet of when the Processor Removed happens:
2024-12-09 12:43:50 processors REDACTEDHOST1 Processor Removed: hr 196613 QEMU Virtual version 2.5+ System
2024-12-09 12:43:49 processors REDACTEDHOST1 Processor Removed: hr 196612 QEMU Virtual version 2.5+ System
2024-12-09 12:43:49 processors REDACTEDHOST1 Processor Removed: hr 196611 QEMU Virtual version 2.5+ System
2024-12-09 12:43:49 processors REDACTEDHOST1 Processor Removed: hr 196610 QEMU Virtual version 2.5+ System
2024-12-09 12:43:49 processors REDACTEDHOST1 Processor Removed: hr 196609 QEMU Virtual version 2.5+ System
2024-12-09 12:43:48 processors REDACTEDHOST1 Processor Removed: hr 196608 QEMU Virtual version 2.5+ System
2024-12-09 12:43:01 processors REDACTEDHOST2 Processor Removed: hr 196631 Intel Xeon E5-2667 v2 @ 3.30GHz System
2024-12-09 12:43:01 processors REDACTEDHOST2 Processor Removed: hr 196630 Intel Xeon E5-2667 v2 @ 3.30GHz System
2024-12-09 12:43:01 processors REDACTEDHOST2 Processor Removed: hr 196629 Intel Xeon E5-2667 v2 @ 3.30GHz System
2024-12-09 12:43:01 processors REDACTEDHOST2 Processor Removed: hr 196628 Intel Xeon E5-2667 v2 @ 3.30GHz System
2024-12-09 12:43:00 processors REDACTEDHOST2 Processor Removed: hr 196627 Intel Xeon E5-2667 v2 @ 3.30GHz System
2024-12-09 12:43:00 processors REDACTEDHOST2 Processor Removed: hr 196626 Intel Xeon E5-2667 v2 @ 3.30GHz
To be clear I filtered the Eventlog for “removed” so if you want other details from it, do let me know.
Yeah I’m still getting this problem to this day.
When I reboot the VMs libreNMS runs discovery like 20-30 seconds after they come back online. I can tell because there’s now a metric on the Host page for each saying “Last Discovery” and listing time, and comparing that to uptime is how I derive the 20-30 second ish difference.
Clearly discovery is happening shortly after each device has booted… why on earth would the seemingly exact same function (discovery) produce “remove CPUs” 20-30 seconds after boot, but if I tell libreNMS to rediscover the exact same host manually… then it sees the CPUs and re-adds them???
This doesn’t make a lick of sense.