Switch Ports and Alerting

I’m fairly certain that I’ve read all the posts / github questions - I may not have understood all the implications of the answers, though.

I have many switches, and of course many ports on each switch.
I want to be alerted once every time a port goes down, but not every X minutes telling me that the down port is still down. Getting a list of all ports that are down at that point isn’t optimal, but it would be OK
And I really, really need the ifAlias in the email, so just parsing the event log by itself isn’t good either.

For recovery mails, I’m fine with a “got better” email every time a down port comes back up, and a “resolved” email when all the monitored ports on a device are back up. Of course it would be awesome to only see the individual port that’s up/down.

I’m currently running 1.37-41-g4da272f and I’m not using the future/in-test-now new alert rule builder. Don’t know if that would help.

At this point I’m utterly helpless and confused. I’ve tried so many things, including OperStatus_prev (which didn’t do what I though it would, but it’s useful in what it does :wink: ), fiddling with max_alerts/delay/interval, even trying to mess with alerted/worse/better status fields (but I probably did that wrong anyway)

Having looked at other people’s posts, I’m afraid: do I really have to have a separate alert rule for each and every port? For instance “operstatus=down AND ifIndex=12” and a “operstatus=down AND ifindex=13” etc.?

If so:

  • would wildcards somewhere help manage things?
  • would/could I add my 200+ (yay, stacks…) port rules straight into the alert_rules database table? Plus another handful of rules depending on other switch vendor’s port naming/numbering conventions.
  • What’s the effect on performance if I have 200+ rules to cover the stack, but my small 24-port switch is now being tested against 180 non-existing ports?
  • is there a better way to skin that cat?

Any help would be greatly appreciated,

  • Frank (aka HerrDrache in discord)

Just set max alerts as 1?

or check the ifOperStatus_prev != ifOperStatus so that it only does it on a change.

“Max Alerts = 1” alerts me when the first port goes down on a switch. Then it has alerted me and won’t alert again when the 2nd port goes down.
Max = -1 alerts me every time the interval is over.

ifOperStatus_prev is a weird one. It doesn’t get updated on every poll (which I mistakenly thought / hoped for), but only on (re)discovery and when the port status changes. So if a port is up and then goes down, ‘prev’ will be ‘up’ for as long as the port stays down. Once the port comes back up, ‘prev’ will be ‘down’ and stay ‘down’ until the port status changes again.

At least that’s what I see with %macros.port_down = 1 && %ports.ifOperStatus_prev = “up” (and variations thereof, including %ports_now_down and friends)

Are you asking a question there or just saying how it is?

You’re correct on the _prev usage, it’s not there to show you previous poll status of the port rather than last status we detected.

Stating how it is and still wondering how I can get a single alert notification on ‘second port is down’. Is there a test that I can use in the alert rule for “device now has different alarms / got worse from last poll”?

I really appreciate your help and time!

I’m an idiot.
LibreNMS monitors devices, once a port goes down, the device is in an “alarm” state, and that doesn’t change if/when additional ports go down.

My solution: Make an alert rule, assign it to devices of interest, mute it so it doesn’t notify via email etc.
Then write a separate script that polls alerts, devices, other magic and that script then can query individual port states and notifies on every individual port up/down change - of course the script has to do the notification itself.

I just hope I don’t need to do that for interface-errors and throughput alerts as well, but that might be where “got better/worse” comes in. Or graphite/grafana.

Sorry for having wasted people’s time!

Is there any way to change this horrible default within LibreNMS?

Not that I know of - but I can share my helper script. It’s probably ugly code by anyone’s standards, but it gets the job done for me. Let me know if you’re interested.

  • Frank

Hey! I’m very interested!

Hey Frank, would also be interested in this as we’re facing the same issues!

Frank, Interested in getting the script too

Chris

Hey Frank, I’m also interested in your script.
Thanks!

Hello Frank, I’m also interested in your script

Regards
Tobias