25.8.0 update corrupted sensor thresholds

Help
1

Hi All,

Overnight our system updated automatically to 25.8.0, however this morning I was greeted with dozens of sensor alerts from various systems, both high temperature and low temperature alerts, as well as voltage alerts…

Looking at things it seems the update has reindexed sensor entities such that sensors and their previous high/low limits are now mismatched. :frowning:

Some examples:

A 3.3v battery sensor “BB +3.3V Vbat” now has limits of high, 13.696, high warn, 13.311, low warn 11.001, low 10.616, so is triggering a low voltage alert. These limits are appropriate for a 12v voltage sensor which is an adjacent sensor in the sensor list.

“P1 DTS Therm Mgn”, which is normally a negative number around -53C (a lot of margin values are negative) suddenly has high and low limits of 115C and 0C so is also alerting.

A final example is “Exit Air Temp” of a server, current value is 24C which is perfectly normal, but the high and low limits are suddenly +10C for the high limit and -66C for the low limit!

I have dozens more. Basically it appears the sensor reported values are correct but all the high and low sensor limit values have played musical chairs among all the sensors and are no longer applied to the correct sensors which has effectively completely broken sensor limit reporting as nothing I’ve looked at has sane high and low limits anymore.

At this point I don’t even know how to recover from this as it’s infeasible to manually edit hundreds of sensor limits, and I have had to disable sensor limit alerts for now as I’m getting frequent alerts from sensors tipping in and out of their new incorrect limits.

Is there some way to “reset” all the sensors back to sensible high and low limit defaults and start again ? While I had manually customised a few sensor limits I can redo those few.

Edit: I’ve also noticed that some of the temperature sensors I have linked directly to graphs on a dashboard have changed and now point to the wrong sensor. For example, this graph linked from the dashboard via a direct URL in an image widget was previously “Front Panel Temp” on a server, but now the same sensor ID is a different sensor, “HSBP 1 Temp”:

image
image788×669 79.6 KB

Looking at the server, these are adjacent sensors in the list, so the sensor index seems to have been offset by one:

image
image812×209 6.96 KB

So now the URL which used to link to “Front Panel Temp” links to “HSBP 1 Temp”.

2

I am having this issue as well. Primarily with SuperMicro IPMI sensors. Dozens of devices throwing sensor out range warnings following 25.8.0 update. The only fix I have found is manually editing thresholds.

3

Hi John,

I ended up using a MySQL query to clear all the sensor limits, then did a rediscovery to repopulate them with automatically calculated values, then tweaked the few that I had manually customised previously.

I think this is the query I used, but I had looked at more than one query and didn’t document exactly which one I used:

USE librenms;

UPDATE sensors
SET sensor_limit = NULL,
    sensor_limit_warn = NULL,
    sensor_limit_low = NULL,
    sensor_limit_low_warn = NULL
WHERE sensor_class IN ('temperature','voltage');

I only cleared temperature and voltage sensor limits, you might want to clear more sensor categories. Maybe someone with more LibreNMS database knowledge than me can confirm if the above is correct.

After that I did a manual rediscovery, which repopulated sensor limits with sensible defaults:

./discovery.php -h all -m sensors

Of course make a database backup before considering running a bulk database update like this! I’m not responsible if this eats your data…

1 Like
4

Thanks for figuring this out! We also faced this for some voltage sensors which now seem to have wrong low limits.