Duplicate Email Alerts

In the last week or so, we are receiving duplicate alert emails for the same rule. Usually there’s 3 seconds difference in the received mail timestamp. Removing the Email Transports and re-adding did not resolve. Duplication also happens on recovery emails. Email Transport setup using our GSuite account

Manually triggered a rule to debug the ./test-alert.php -r 15 -d -h 46. It appears to trigger “:: Transport mail” twice. Output of ./validate.php & ./test-alert.php below. Any help is appreciated!

Validate:

ubuntu@nms:/opt/librenms$ sudo ./validate.php
====================================
Component | Version
--------- | -------
LibreNMS  | 1.48.1-76-g914531a28
DB Schema | 2019_02_10_220000_add_dates_to_fdb (132)
PHP       | 7.2.15-0ubuntu0.18.04.1
MySQL     | 10.1.38-MariaDB-0ubuntu0.18.04.1
RRDTool   | 1.7.0
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.8.4
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct

Test-Alert:

/opt/librenms/scripts$ sudo ./test-alert.php -r 15 -d -h 46
SQL[SELECT alerts.id, alerts.device_id, alerts.rule_id, alerts.state, alerts.note, alerts.info FROM alerts WHERE alerts.device_id = 46 && alerts.rule_id = 15 [] 0.21ms]

SQL[SELECT alert_log.id,alert_log.rule_id,alert_log.device_id,alert_log.state,alert_log.details,alert_log.time_logged,alert_rules.rule,alert_rules.severity,alert_rules.extra,alert_rules.name,alert_rules.builder FROM alert_log,alert_rules WHERE alert_log.rule_id = alert_rules.id && alert_log.device_id = ? && alert_log.rule_id = ? && alert_rules.disabled = 0 ORDER BY alert_log.id DESC LIMIT 1 [46,15] 0.24ms]

SQL[SELECT DISTINCT a.* FROM alert_rules a
  LEFT JOIN alert_device_map d ON a.id=d.rule_id
  LEFT JOIN alert_group_map g ON a.id=g.rule_id
  LEFT JOIN device_group_device dg ON g.group_id=dg.device_group_id
  WHERE a.disabled = 0 AND ((d.device_id IS NULL AND g.group_id IS NULL) OR d.device_id=? OR dg.device_id=?) [46,46] 0.22ms]

SQL[SELECT attrib_value FROM devices_attribs WHERE attrib_type = "disable_notify" && device_id = ? [46] 0.14ms]

SQL[SELECT hostname, sysName, sysDescr, sysContact, os, type, ip, hardware, version, purpose, notes, uptime, status, status_reason, locations.location FROM devices LEFT JOIN locations ON locations.id = devices.location_id WHERE device_id = ? [46] 0.35ms]

SQL[SELECT * FROM devices_attribs WHERE `device_id` = ? [46] 0.13ms]

SQL[select * from `device_perf` where `device_id` = ? order by `timestamp` desc limit 1 [46] 2.96ms]

SQL[select * from `alert_templates` where exists (select * from `alert_template_map` where `alert_templates`.`id` = `alert_template_map`.`alert_templates_id` and `alert_rule_id` = ?) limit 1 [15] 0.19ms]

Issuing Alert-UID #6450/1:
SQL[SELECT `rule_id` FROM `alerts` WHERE `id`=? [419] 0.14ms]

SQL[SELECT b.transport_id, b.transport_type, b.transport_name FROM alert_transport_map AS a LEFT JOIN alert_transports AS b ON b.transport_id=a.transport_or_group_id WHERE a.target_type='single' AND a.rule_id=? UNION DISTINCT SELECT d.transport_id, d.transport_type, d.transport_name FROM alert_transport_map AS a LEFT JOIN alert_transport_groups AS b ON a.transport_or_group_id=b.transport_group_id LEFT JOIN transport_group_transport AS c ON b.transport_group_id=c.transport_group_id LEFT JOIN alert_transports AS d ON c.transport_id=d.transport_id WHERE a.target_type='group' AND a.rule_id=? [15,15] 0.26ms]

 :: Transport mail => SQL[SELECT `transport_config` FROM `alert_transports` WHERE `transport_id`=? [2] 0.13ms]

Attempting to email [NMS] warning - Host Down - 34.241.80.78 to: [email protected]
OKSQL[SELECT `devices`.*, `location`, `lat`, `lng` FROM `devices` LEFT JOIN locations ON `devices`.location_id=`locations`.`id` WHERE `device_id` = ? [46] 0.56ms]

SQL[SELECT * FROM devices_attribs WHERE `device_id` = ? [46] 0.2ms]

SQL[SELECT * FROM `vrf_lite_cisco` WHERE `device_id` = ? [46] 4.98ms]

SQL[INSERT IGNORE INTO `eventlog` (`device_id`,`reference`,`type`,`datetime`,`severity`,`message`,`username`)  VALUES (:device_id,:reference,:type,:datetime,:severity,:message,:username) {"device_id":46,"reference":null,"type":"alert","datetime":"2019-02-21 14:45:15","severity":1,"message":"Issued warning alert for rule 'Device Status' to transport 'mail'","username":""} 2.13ms]


 :: Transport mail => Attempting to email [NMS] warning - Host Down - 34.241.80.78 to: [email protected]
OKSQL[INSERT IGNORE INTO `eventlog` (`device_id`,`reference`,`type`,`datetime`,`severity`,`message`,`username`)  VALUES (:device_id,:reference,:type,:datetime,:severity,:message,:username) {"device_id":46,"reference":null,"type":"alert","datetime":"2019-02-21 14:45:17","severity":1,"message":"Issued warning alert for rule 'Device Status' to transport 'mail'","username":""} 2.19ms]

Crontab:

# Using this cron file requires an additional user on your system, please see install docs.

33   */6  * * *   librenms    /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1
*/5  *    * * *   librenms    /opt/librenms/discovery.php -h new >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/cronic /opt/librenms/poller-wrapper.py 16
*    *    * * *   librenms    /opt/librenms/alerts.php >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/poll-billing.php >> /dev/null 2>&1
01   *    * * *   librenms    /opt/librenms/billing-calculate.php >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/check-services.php >> /dev/null 2>&1

# Daily maintenance script. DO NOT DISABLE!
# If you want to modify updates:
#  Switch to monthly stable release: https://docs.librenms.org/General/Releases/
#  Disable updates: https://docs.librenms.org/General/Updating/
15   0    * * *   librenms    /opt/librenms/daily.sh >> /dev/null 2>&1
2 Likes

Wondering if anyone has any suggestions on where to start looking? Issue is still happening.
There is another (older) post about duplicate smtp mail alerts but that seems specific to distributed poller setups (post).

Validate:

|Version|1.48.1-81-g48ed66596 - Sun Feb 24 2019 22:02:44 GMT+0000|
|---|---|
|DB Schema|2019_02_10_220000_add_dates_to_fdb (132)|
|Web Server|Apache/2.4.29 (Ubuntu)|
|PHP|7.2.15-0ubuntu0.18.04.1|
|MySQL|10.1.38-MariaDB-0ubuntu0.18.04.1|
|RRDtool|1.7.0|

If any other output is required, please let me know. Any assistance is much appreciated!

1 Like

I believe this is issue with new transports when we moved from the old transports. The old ones are still in the DB. A fix hasn’t been created.

is there a fix soon for this issue?

I don’t know nobody has submitted one.

https://github.com/librenms/librenms/pulls

I opened one as I have the same issue, if you can contribute more details that would be great:

Please dont open issues in the git thats only for new device requests. Please keep it here or in discord.

Same here, duplicate mails for alerts and recovery… Same version, and DB Schema.

I disabled email alerting in Global Settings -> email options.

1 Like

Thanks JohnSPeach! This fixed it for me too:

Yes this worked for me too!

I wouldn’t classify the complete disabling of email alerting as a fix. To be sure, it does make all email alerts go away (completely!), but that’s a problem too, because I need email alerting to work - without filling up my inbox with duplicates.

Disabling General Email Alerting will not disable email alerting at all. Will only disable that section.

Email transports should work without issues

Well, I stand corrected! The first attempt at disabling general email alerting seemed to kill all alerting. I left it that way overnight, and I did, in fact, get some alerts. Thanks!