Currently all alert would be received in a single channel. Slack has (un)fortunately become the centralised place for all information. We currently got channels for notifications (low impact, looked after by junior staff) and alerts (high impact, managers like to keep an eye out in here).
Based on the channel, slack would alert + send appropriate push notifications to different people.
+1 for this. ie: Grafana
You can choose various notification “channels”, and for each you can select the transport, and various specifics.
If the transport.slack.php could also grab Alert Templates pre-assigned to each alert rule, this could possibly accomplish this? Then the template would just need to have “@mentions” and preferably the “channel” call?
Assuming you had that functionality and you set a channel for critical and a channel for warning messages, where would you want the ok status messages to appear when the service returns to normal operation?
Please note that the Transport will have no memory on where the previous messages were sent too.
If Alert rule X exists, and its severity is set to “warning” and I have a slack transport called “slack_warning” and the alert rule X transport is set to use “slack_warning” for transport, then all the warning alerts for alert rule X are sent to that slack channel, as well as the recovery alerts.
You could also create a transport group and send both the alert and recovery notifications to multiple slack channels at once.
I feel like this functionality is already present in the current release of LibreNMS.
This would require that the alert sub system stores in the database which alert was sent to which channel, also requires multiple lookups on the DB for each alert, this would not be an issue on a small installation but it can and will create a lot of load on installations with a few hundred (or more) alerts per hour.
If the alert logic understands that it only sends an “Ok” alert if the previous status was something other than “Ok”, the previous status must be tracked somewhere. Could it not select transport based on that?