In the last week or so, we are receiving duplicate alert emails for the same rule. Usually there’s 3 seconds difference in the received mail timestamp. Removing the Email Transports and re-adding did not resolve. Duplication also happens on recovery emails. Email Transport setup using our GSuite account
Manually triggered a rule to debug the ./test-alert.php -r 15 -d -h 46. It appears to trigger “:: Transport mail” twice. Output of ./validate.php & ./test-alert.php below. Any help is appreciated!
Validate:
ubuntu@nms:/opt/librenms$ sudo ./validate.php
====================================
Component | Version
--------- | -------
LibreNMS | 1.48.1-76-g914531a28
DB Schema | 2019_02_10_220000_add_dates_to_fdb (132)
PHP | 7.2.15-0ubuntu0.18.04.1
MySQL | 10.1.38-MariaDB-0ubuntu0.18.04.1
RRDTool | 1.7.0
SNMP | NET-SNMP 5.7.3
====================================
[OK] Composer Version: 1.8.4
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
Test-Alert:
/opt/librenms/scripts$ sudo ./test-alert.php -r 15 -d -h 46
SQL[SELECT alerts.id, alerts.device_id, alerts.rule_id, alerts.state, alerts.note, alerts.info FROM alerts WHERE alerts.device_id = 46 && alerts.rule_id = 15 [] 0.21ms]
SQL[SELECT alert_log.id,alert_log.rule_id,alert_log.device_id,alert_log.state,alert_log.details,alert_log.time_logged,alert_rules.rule,alert_rules.severity,alert_rules.extra,alert_rules.name,alert_rules.builder FROM alert_log,alert_rules WHERE alert_log.rule_id = alert_rules.id && alert_log.device_id = ? && alert_log.rule_id = ? && alert_rules.disabled = 0 ORDER BY alert_log.id DESC LIMIT 1 [46,15] 0.24ms]
SQL[SELECT DISTINCT a.* FROM alert_rules a
LEFT JOIN alert_device_map d ON a.id=d.rule_id
LEFT JOIN alert_group_map g ON a.id=g.rule_id
LEFT JOIN device_group_device dg ON g.group_id=dg.device_group_id
WHERE a.disabled = 0 AND ((d.device_id IS NULL AND g.group_id IS NULL) OR d.device_id=? OR dg.device_id=?) [46,46] 0.22ms]
SQL[SELECT attrib_value FROM devices_attribs WHERE attrib_type = "disable_notify" && device_id = ? [46] 0.14ms]
SQL[SELECT hostname, sysName, sysDescr, sysContact, os, type, ip, hardware, version, purpose, notes, uptime, status, status_reason, locations.location FROM devices LEFT JOIN locations ON locations.id = devices.location_id WHERE device_id = ? [46] 0.35ms]
SQL[SELECT * FROM devices_attribs WHERE `device_id` = ? [46] 0.13ms]
SQL[select * from `device_perf` where `device_id` = ? order by `timestamp` desc limit 1 [46] 2.96ms]
SQL[select * from `alert_templates` where exists (select * from `alert_template_map` where `alert_templates`.`id` = `alert_template_map`.`alert_templates_id` and `alert_rule_id` = ?) limit 1 [15] 0.19ms]
Issuing Alert-UID #6450/1:
SQL[SELECT `rule_id` FROM `alerts` WHERE `id`=? [419] 0.14ms]
SQL[SELECT b.transport_id, b.transport_type, b.transport_name FROM alert_transport_map AS a LEFT JOIN alert_transports AS b ON b.transport_id=a.transport_or_group_id WHERE a.target_type='single' AND a.rule_id=? UNION DISTINCT SELECT d.transport_id, d.transport_type, d.transport_name FROM alert_transport_map AS a LEFT JOIN alert_transport_groups AS b ON a.transport_or_group_id=b.transport_group_id LEFT JOIN transport_group_transport AS c ON b.transport_group_id=c.transport_group_id LEFT JOIN alert_transports AS d ON c.transport_id=d.transport_id WHERE a.target_type='group' AND a.rule_id=? [15,15] 0.26ms]
:: Transport mail => SQL[SELECT `transport_config` FROM `alert_transports` WHERE `transport_id`=? [2] 0.13ms]
Attempting to email [NMS] warning - Host Down - 34.241.80.78 to: [email protected]
OKSQL[SELECT `devices`.*, `location`, `lat`, `lng` FROM `devices` LEFT JOIN locations ON `devices`.location_id=`locations`.`id` WHERE `device_id` = ? [46] 0.56ms]
SQL[SELECT * FROM devices_attribs WHERE `device_id` = ? [46] 0.2ms]
SQL[SELECT * FROM `vrf_lite_cisco` WHERE `device_id` = ? [46] 4.98ms]
SQL[INSERT IGNORE INTO `eventlog` (`device_id`,`reference`,`type`,`datetime`,`severity`,`message`,`username`) VALUES (:device_id,:reference,:type,:datetime,:severity,:message,:username) {"device_id":46,"reference":null,"type":"alert","datetime":"2019-02-21 14:45:15","severity":1,"message":"Issued warning alert for rule 'Device Status' to transport 'mail'","username":""} 2.13ms]
:: Transport mail => Attempting to email [NMS] warning - Host Down - 34.241.80.78 to: [email protected]
OKSQL[INSERT IGNORE INTO `eventlog` (`device_id`,`reference`,`type`,`datetime`,`severity`,`message`,`username`) VALUES (:device_id,:reference,:type,:datetime,:severity,:message,:username) {"device_id":46,"reference":null,"type":"alert","datetime":"2019-02-21 14:45:17","severity":1,"message":"Issued warning alert for rule 'Device Status' to transport 'mail'","username":""} 2.19ms]
Crontab:
# Using this cron file requires an additional user on your system, please see install docs.
33 */6 * * * librenms /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1
*/5 * * * * librenms /opt/librenms/discovery.php -h new >> /dev/null 2>&1
*/5 * * * * librenms /opt/librenms/cronic /opt/librenms/poller-wrapper.py 16
* * * * * librenms /opt/librenms/alerts.php >> /dev/null 2>&1
*/5 * * * * librenms /opt/librenms/poll-billing.php >> /dev/null 2>&1
01 * * * * librenms /opt/librenms/billing-calculate.php >> /dev/null 2>&1
*/5 * * * * librenms /opt/librenms/check-services.php >> /dev/null 2>&1
# Daily maintenance script. DO NOT DISABLE!
# If you want to modify updates:
# Switch to monthly stable release: https://docs.librenms.org/General/Releases/
# Disable updates: https://docs.librenms.org/General/Updating/
15 0 * * * librenms /opt/librenms/daily.sh >> /dev/null 2>&1