Sensor Limits Changing "Automatically"

Tags: #<Tag:0x00007f6cb2c0da98>
#1

OK, it seems like I’m going crazy, but this has happened too many times for that :laughing:. I manually change some sensor / health limits, as I want (e.g. Vcore for my CPU) … and then a day or two later, they are reset (automatically it seems). So I change them again, and the cycle repeats.

Is there some setting that is enabing this override of my manual settings?

Thanks!

#2

Just curious - anyone else seeing this? For my storage devices in particular, it’s happening almost daily.

Thanks!

#3

It does not happen in general, but sometimes, with certain devices/OS, the sensors are deleted and created again. In that case you will loose any manual setting.
Could you check that your sensors are not deleted/recreated ?
If yes, the step coming next is to understand why and fix it :smiley:

#4

Yes, I do think that’s it - I say that because the (Ubuntu Linux) server that this is running on … well, the other day I happened to notice that it had “lost” all it’s drives (in LibreNMS), except for two NFS mounts. For example, even the root ("/") partition was “gone”. Not sure why it is, but I do see devices and sensors dropping on this machine - and it’s the LibreNMS server, so clearly up :grinning_face_with_smiling_eyes:.

Suggestions of things to check?

Thanks!

#5

You’re right! Got the warning again today (custom values lost), and I checked that device. A whole bunch of these (all sensors I think),

image
image1677×310 13.4 KB

Open to any suggestions. Thanks!

#6

Do you have any CPU overloading on your LibreNMS server ? Or on the device being monitored ? Any high latency ?
I have this kind of behaviour with an old Mac Mini running Debian, where the sensors are changing their ID randomly during reboot. So LibreNMS discovers new sensors after 50% of reboots.

#7

I don’t think so (could be wrong of course). The server and machine being monitored are the same (though I have seen this with other “clients” as well). As for overload, here is top output,

top - 13:51:36 up 3 days, 17:36,  3 users,  load average: 1.18, 1.08, 1.00
Tasks: 509 total,   1 running, 508 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  1.4 sy,  0.0 ni, 98.2 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  32102.4 total,    870.8 free,  19688.7 used,  11542.9 buff/cache
MiB Swap:  15320.0 total,   7670.0 free,   7650.0 used.  12310.6 avail Mem

So I don’t think it’s overloaded - it’s a Ryzen 7 2700X, and as you see ~ 98% idle :grinning_face_with_smiling_eyes:.

Thanks!

#8

yep. Looks good. I don’t see why these sensors are removed/rediscovered. I guess you’ll have to dig into the code to find out …

#9

Yep, agreed. As this happens very infrequently (i.e. can go a few weeks between occurrences) - I need to figure out how to get logging beefed up, to help debug.

Thanks!

#10

Arrgh - happened again today … and no OS updates, not even a reboot. Only “change” is the daily update to LibreNMS - I don’t see sensors going away / being re-added, but custom settings are reset again.

Is there a way to have debug output captured for all polls (for a single device)? Just asking because it’s very random, really need to capture it all to debug.

Thanks!