We recently upgraded LibreNMS from v24.3.0 to v25.5.0 and have since observed that Syslog alert rules are generating multiple “Has worsened” notifications—sometimes just milliseconds apart—for the same issue. Initially, we suspected duplicate Syslog messages, but we’ve confirmed that this is not the case.
It appears that even when a matching alert is already active, additional matching Syslog messages continue to trigger new “Has worsened” notifications, resulting in alert floods.
Any assistance or best practices for tuning the alerts to avoid redundant “Has worsened” alerts would be greatly appreciated.
===========================================
Component | Version
--------- | -------
LibreNMS | 25.5.0 (2025-05-16T14:23:44-07:00)
DB Schema | 2025_05_03_152418_remove_invalid_sensor_classes (340)
PHP | 8.3.19
Python | 3.12.10
Database | MariaDB 10.5.19-MariaDB-1:10.5.19+maria~ubu2004
RRDTool | 1.9.0
SNMP | 5.9.4
===========================================
[OK] Installed from package; no Composer required
[OK] Database Connected
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK] Database and column collations are correct
[OK] Database schema correct
[OK] MySQL and PHP time match
[OK] Distributed Polling setting is enabled globally
[OK] Connected to rrdcached
[OK] Active pollers found
[OK] Dispatcher Service is enabled
[OK] Locks are functional
[OK] No python wrapper pollers found
[OK] Redis is functional
[WARN] IPv6 is disabled on your server, you will not be able to add IPv6 devices.
[OK] rrdtool version ok
[OK] Connected to rrdcached
[WARN] Non-git install, updates are manual or from package
The OR statement is nested in an AND statement. Here is SQL query for context
SELECT * FROM devices,syslog WHERE (devices.device_id = ? AND devices.device_id = syslog.device_id) AND syslog.timestamp >= (DATE_SUB(NOW(),INTERVAL 5 MINUTE)) AND syslog.msg LIKE '%FULL to DOWN%' AND (syslog.program LIKE '%OSPF-%' OR syslog.msg LIKE '%OSPF-%')
We did some additional testing and noticed that this issue is related to the “Interval” setting. According to the tooltip, setting the Interval to 0 should result in only one notification being sent while the alert remains active. However, that doesn’t appear to be working as intended.
As shown in the screenshot, the initial alert notification is sent at 16:02:58 and is followed by many “Has worsened” notifications for the same alert.
As a workaround we changed Interval to 5 minutes and the repeated notifications stopped.
Could this be a bug related to the Interval setting?
you are using interval as a way to delay the alert (delay would work the same probably).
You should run your raw SQL query against your database. I’m almost certain it isn’t working the way you expect.
Here is a tip when an alert is triggered, it records the rows returned by the query. If those rows change later, you get a got worse or got better notification.
Honestly, I think your alert is triggering itself
FYI, This is my OSPF alert rule: *macros.device_up = 1 AND ospf_nbrs.ospfNbrState NOT REGEXP "(full|twoWay)"*
Unfortunately, I do get alerts from all neighbors when one device goes down.
Appreciate the tip - I can do some further testing on the query to confirm that.
What’s throwing me off though is that this wasn’t happening before upgrading to v25.5.0. The alert config and query stayed the same but the notification behavior changed right after the upgrade. So I’m wondering if something changed in how alerts are handled now?
The goal of using alert interval set to 0 was that when a device sent OSPF down syslog, we would get one notification only, even if the link flapped a few times within the 5 min query window. After the version upgrade, without any change to the alert rule, we are now getting the “better” or “worse” alerts based on count of syslog messages within the last 5 minutes.
After some testing, it looks like there may be a bug or behavior change with interval=0. When set to 0, it is still sending multiple worse/better alerts before recovering.
