Re-Alerting When Event is Triggered Again

Hi,

I’m evaluating LibreNMS for my organization and so far it’s been great, but there are a few cases where LibreNMS alerts doesn’t fully suite our needs or makes it painful to suit our needs.

For example, I’ve configured SNMP traps for all my devices, and I have a alert rule that forwards traps to our Slack channel. However, this rule has an interval, and only alerts us after the first trap is generated.

Let’s say at 12:01 I receive a trap for a PSU going down, Ill get alerted via Slack which is perfect and this fits our needs.

However, at 12:15 I receive a trap for bpduguard being activated. Because of the interval of 1 hour, I will not get alerted immediately and therefore won’t respond to a pressing issue. I could make the interval shorter, but this only increases noise, especially if the issue takes a while to resolve. Basically, if a new event triggers the same alert, I would like to get notified, but if its the same event, I would like it to stay silent.

I would love it if there were a feature that would allow us to get notified for an already acknowledge alert if something changes. My greatest concern regarding the lack of this feature is not being notified during a double circuit failure. Because of this, I have an individual rule for all my circuits and igp/bgp sessions. However, this is not a scalable solution as we grow and it becomes incredibly painful to maintain.

I appreciate the work that everyone put LibreNMS and this is a truly amazing product.

Use the ACK button with lower delay, if you ack an alert you wont recieve any notification and it will retrigger if another event on that alert happens, for better (issue solved) or for worse.

Hi,

Thanks for the response!

I’m not quite sure what you mean by “ACK with lower delay” When I click the ACK button, the only option is I see is “Acknowledge until clear” which is currently off.

In my alert rules, I have a delay of 1 minute, max alerts of -1, and interval of 1 hour. However, when I am sending fake traps to LibreNMS for testing, I am sending them with several minute intervals, but I don’t receive alerts for traps sent after the first one. I am sending a different trap message each time during testing.

EDIT: I just noticed that when I send a different trap message after acknowledging the previous, the notification in LibreNMS is unacknowledged and I have to acknowledge it again. However, the transport is not triggered when this happens.

Oops, sorry, Missread delay when you said interval.

What makes me think if a new trigger should wait for the interval between notifications…

Also, what I said still correct. If you set a lower interval and ack the alert, you wont be notified until clear/worse/better status on that alert.

I should’ve mentioned earlier that I’m concerned about noise because we will have an integration with PagerDuty, and I’m worried about blowing up my co-workers phone at 3 AM with a million calls regarding 1 alert.

I think I figured out how to accomplish what I want. I’ve changed my alerts with the following settings:

Max alerts = 1
Delay = 1m
Interval = 1m
Transport = slack/pagerduty/email

Correct me if I’m wrong, but with these settings I’ll have:

  • One alert per event with no subsequent notifications (no multiple slack/pagerduty/email)
  • Another alert only if something changes (recovery/worse)
  • If something changes within that minute interval, I’ll miss it. e.g alert 1 comes at 12:00:01, alert 2 comes at 12:00:02