Yesterday I posted the beginnings of this on discord, but according to discord, today’s reply was too large to send there, so I’ll re-submit the whole thread here.
The main issue was it seemed LibreNMS was alerting on a BGP rule because State was empty, when drilling down into the alert, once it’s fired:
Libre > Alerts > Notications > +
#1: BGP peer 10.0.20.1, AS65001, State
#2: BGP peer 10.0.21.1, AS65001, State
The alert was a slightly modified variant of the built-in BGP Session down rule:
SELECT * FROM devices,bgpPeers WHERE (devices.device_id = ? AND devices.device_id = bgpPeers.device_id) AND bgpPeers.bgpPeerState != "established" AND (devices.status = 1 && (devices.disabled = 0 && devices.ignore = 0)) = 1 AND bgpPeers.bgpPeerAdminStatus != "stop" AND bgpPeers.bgpPeerRemoteAs = 65001
Despite State in the alert drill-down being empty, snmpwalk reports bgpPeerState and bgpPeerAdminStatus just fine:
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerAdminStatus.10.0.20.1 = INTEGER: start(2)
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerAdminStatus.10.0.21.1 = INTEGER: start(2)
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerState.10.0.20.1 = INTEGER: established(6)
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerState.10.0.21.1 = INTEGER: established(6)
Today, I’ve done some more digging and it seems this may be related to SNMPv3 specifically.
device_id 10 is the device whose output I posted yesterday. device_id 11 is some other device that also has BGP sessions matched (and erroneously alerting) against the same rule:
Here, both are configured to use SNMPv3 but the State/Status is empty:
MariaDB [librenms]> SELECT devices.overwrite_ip,snmpVer,bgpPeerState,bgpPeerAdminStatus FROM devices,bgpPeers WHERE (devices.device_id = 10 OR devices.device_id = 11) AND devices.device_id = bgpPeers.device_id AND bgpPeers.bgpPeerRemoteAs = 65001;
+----------------+---------+--------------+--------------------+
| overwrite_ip | snmpVer | bgpPeerState | bgpPeerAdminStatus |
+----------------+---------+--------------+--------------------+
| 192.168.57.107 | v3 | | |
| 192.168.57.107 | v3 | | |
| 192.168.57.111 | v3 | | |
| 192.168.57.111 | v3 | | |
+----------------+---------+--------------+--------------------+
4 rows in set (0.001 sec)
MariaDB [librenms]>
If I reconfigure Libre to poll device_id 10 using SNMPv2 instead of SNMPv3, bgpPeerState, bgpPeerAdminStatus show the actual state, and the alarm for that device clears.
MariaDB [librenms]> SELECT devices.overwrite_ip,snmpVer,bgpPeerState,bgpPeerAdminStatus FROM devices,bgpPeers WHERE (devices.device_id = 10 OR devices.device_id = 11) AND devices.device_id = bgpPeers.device_id AND bgpPeers.bgpPeerRemoteAs = 65001;
+----------------+---------+--------------+--------------------+
| overwrite_ip | snmpVer | bgpPeerState | bgpPeerAdminStatus |
+----------------+---------+--------------+--------------------+
| 192.168.57.107 | v2c | established | start |
| 192.168.57.107 | v2c | established | start |
| 192.168.57.111 | v3 | | |
| 192.168.57.111 | v3 | | |
+----------------+---------+--------------+--------------------+
4 rows in set (0.001 sec)
MariaDB [librenms]>
For posterity, manually polling the devices using v3 for device_id 10 and device_id 11:
bash-5.0# snmpwalk -v3 -l authPriv -a SHA -A *** -x AES -X *** -u *** -m CISCO-BGP4-MIB -M /opt/librenms/mibs:/opt/librenms/mibs/cisco -Of 192.168.57.107 bgpPeerAdminStatus
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerAdminStatus.10.0.20.1 = INTEGER: start(2)
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerAdminStatus.10.0.21.1 = INTEGER: start(2)
bash-5.0# snmpwalk -v3 -l authPriv -a SHA -A *** -x AES -X *** -u *** -m CISCO-BGP4-MIB -M /opt/librenms/mibs:/opt/librenms/mibs/cisco -Of 192.168.57.107 bgpPeerState
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerState.10.0.20.1 = INTEGER: established(6)
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerState.10.0.21.1 = INTEGER: established(6)
bash-5.0# snmpwalk -v3 -l authPriv -a SHA -A *** -x AES -X *** -u *** -m CISCO-BGP4-MIB -M /opt/librenms/mibs:/opt/librenms/mibs/cisco -Of 192.168.57.111 bgpPeerAdminStatus
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerAdminStatus.10.0.20.1 = INTEGER: start(2)
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerAdminStatus.10.0.21.1 = INTEGER: start(2)
bash-5.0# snmpwalk -v3 -l authPriv -a SHA -A *** -x AES -X *** -u *** -m CISCO-BGP4-MIB -M /opt/librenms/mibs:/opt/librenms/mibs/cisco -Of 192.168.57.111 bgpPeerState
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerState.10.0.20.1 = INTEGER: established(6)
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerState.10.0.21.1 = INTEGER: established(6)
bash-5.0#
Also, manually polling the devices using v2 for device_id 10:
bash-5.0# snmpwalk -v2c -c *** -m CISCO-BGP4-MIB -M /opt/librenms/mibs:/opt/librenms/mibs/cisco -Of 192.168.57.107 bgpPeerAdminStatus
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerAdminStatus.10.0.20.1 = INTEGER: start(2)
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerAdminStatus.10.0.21.1 = INTEGER: start(2)
bash-5.0# snmpwalk -v2c -c *** -m CISCO-BGP4-MIB -M /opt/librenms/mibs:/opt/librenms/mibs/cisco -Of 192.168.57.107 bgpPeerState
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerState.10.0.20.1 = INTEGER: established(6)
.iso.org.dod.internet.mgmt.mib-2.bgp.bgpPeerTable.bgpPeerEntry.bgpPeerState.10.0.21.1 = INTEGER: established(6)
bash-5.0#