Every 10 minutes the switch port variables change

[thaz]

It seems every 10 minutes or so, the switch ports change from NULL to a number, and then back to NULL, over and over… What is the best way to troubleshoot this?

[thaz]

Spent more time to debug this… It seems every time they poller is ran, the ports change from HOSTNAME to NULL, and then the next time, they go NULL → HOSTNAME.

Example:

Port xe-0/1/6: xe-0/1/6 (540 / #6073) ifHCInOctets
ifHCOutOctets
ifHCInUcastPkts
ifHCOutUcastPkts
ifHCInBroadcastPkts
ifHCOutBroadcastPkts
ifHCInMulticastPkts
ifHCOutMulticastPkts
ifHighSpeed (10000)
dot3Duplex VLAN = ?  Using ifDescr as ifAlias
SQL[insert into `eventlog` (`reference`, `type`, `datetime`, `severity`, `message`, `username`, `device_id`) values (?, ?, ?, ?, ?, ?, ?) [6073,"interface","2024-07-22 20:55:13",3,"ifAlias: AZ1-ROUTER1.Ethernet1\/2 -> xe-0\/1\/6","",208] 0.96ms]

ifAlias: AZ1-ROUTER1.Ethernet1/2 -> xe-0/1/6
SQL[insert into `eventlog` (`reference`, `type`, `datetime`, `severity`, `message`, `username`, `device_id`) values (?, ?, ?, ?, ?, ?, ?) [6073,"interface","2024-07-22 20:55:13",4,"ifConnectorPresent: true -> NULL","",208] 1.03ms]

ifConnectorPresent: true -> NULL
xe-0/1/6SQL[insert into `eventlog` (`reference`, `type`, `datetime`, `severity`, `message`, `username`, `device_id`) values (?, ?, ?, ?, ?, ?, ?) [6073,"interface","2024-07-22 20:55:13",3,"type: az-nni -> NULL","",208] 0.99ms]

SQL[insert into `eventlog` (`reference`, `type`, `datetime`, `severity`, `message`, `username`, `device_id`) values (?, ?, ?, ?, ?, ?, ?) [6073,"interface","2024-07-22 20:55:13",3,"descr: AZ1-ROUTER1.Ethernet1\/2 -> NULL","",208] 1.02ms]


 ifInOctets (54043 B) 293.71195652174 Bps 184 secs

 ifOutOctets (930305776 B) 5056009.6521739 Bps 184 secs

 ifInErrors (0 B) 0 Bps 184 secs

 ifOutErrors (0 B) 0 Bps 184 secs

 ifInUcastPkts (625 B) 3.3967391304348 Bps 184 secs

 ifOutUcastPkts (4043992 B) 21978.217391304 Bps 184 secs

 ifInNUcastPkts (0 B) 0 Bps 184 secs

 ifOutNUcastPkts (0 B) 0 Bps 184 secs

 ifInDiscards (0 B) 0 Bps 184 secs

 ifOutDiscards (0 B) 0 Bps 184 secs

 ifInUnknownProtos (0 B) 0 Bps 184 secs

 ifInBroadcastPkts (0 B) 0 Bps 184 secs

 ifOutBroadcastPkts (0 B) 0 Bps 184 secs

 ifInMulticastPkts (29 B) 0.15760869565217 Bps 184 secs

 ifOutMulticastPkts (34 B) 0.18478260869565 Bps 184 secs
bps(2.35 kbps/40.45 Mbps)bytes(52.78 KiB/887.21 MiB)pkts(3.4 pps/21.98 kpps)RRD warning: unused data sent ifInUcastPkts_rate
RRD warning: unused data sent ifOutUcastPkts_rate
RRD warning: unused data sent ifInErrors_rate
RRD warning: unused data sent ifOutErrors_rate
RRD warning: unused data sent ifInOctets_rate
RRD warning: unused data sent ifOutOctets_rate
RRD warning: unused data sent ifInBits_rate
RRD warning: unused data sent ifOutBits_rate
RRD[update /opt/librenms/rrd/69.48.137.240/port-id6073.rrd N:435857374062:43345769661623:0:0:918771927:103526399603:U:U:0:0:U:3:1738:560635:685260]
RRDtool Output: OK u:0.01 s:0.01 r:10.70

[thaz]

I wonder if it’s related to this:

Caching Oids: Full ports polling SNMP['/usr/bin/snmpbulkwalk' '-v2c' '-c' 'xx' '-OQUs' '-m' 'IF-MIB' '-M' '/opt/librenms/mibs:/opt/librenms/mibs/junos' 'udp:1.2.3.4:161' 'ifXEntry']
Exitcode: 1
Timeout: No Response from udp:1.2.3.4:161

What’s weird is that all the other graphs and info seems to be pulling fine, but I do see 2 SNMP timeouts… maybe it just assumes a timeout means it should remove all of the existing data?

[thaz]

I figured it out

The switch had a ddos protection mode that was being activated everytime LibreNMS tried to connect via SNMP. So, when LibreNMS tried to poll the next interval, and it couldn’t reach it - it made all of the entries ‘NULL’. It seems silly to me that it would do that and it wouldn’t just wait until the next time it connected, but it is what it is.

Once I fixed the ddos mitigation, the NULL values stopped.