Weird throughput report on Juniper em3 polling

Hi,

We have an issue that started only last night, where our em3 interfaces on upgraded Juniper QFX10002 switches reports a throughput in the Pbps range and complains about “Port utilisation over threshold”. However, the graphs report only between 15-20Kbps, and when doing manual polling, I have similar results as what the graphs give out.

Any clue as to what could be causing this? The actual throughput on the interface only started to be reported since our latest upgrade, as before that only PPS were being reported by SNMP. However, the upgrade was done about 2 weeks ago and this alert just started last night.

We have another switch of the same model that we have just upgraded last night, and it’s also just started to respond to throughput request, yet hasn’t started to show any issue on the reported throughput as it matches what’s on the graph.

Thanks!

Hi! I was wondering if anyone had a clue on this one? We’ve since upgraded to 1.34, but the issue persists.

Thanks again!

try rrd tune?

https://docs.librenms.org/#Extensions/RRDTune/

make sure to run the scripts also.

Thanks. I just did, enabling the RRDTune globally and ran the script, and still get the same results.

$ ./scripts/tune_port.php -h host -p em3
Found hostname host.......
Tuning port em3.......

I then enabled it explicitly for the interface of one of the other devices that has the issue, ran the script for that one, and no change either.

could be buggy firmware on that device?

If I do manual ifInOctets polling, diff the results and divide by time, I get values that match what’s in the graphs, which is the proper value (very very low traffic). And the Pbps don’t match at all with what’s in the graph, which is much more aligned with what the unit is actually doing. The errors also started just about 13 days after the upgrade, which is even more confusing…

As a reference:

$ snmpwalk -nn -v2c -c IF-MIB::ifName | grep em3
IF-MIB::ifName.152 = STRING: em3
$ snmpwalk -nn -v2c -c IF-MIB::ifInOctets.152; date
IF-MIB::ifInOctets.152 = Counter32: 3382464467
Wed Nov 29 17:02:23 EST 2017
$ snmpwalk -nn -v2c -c IF-MIB::ifInOctets.152; date
IF-MIB::ifInOctets.152 = Counter32: 3382476299
Wed Nov 29 17:02:31 EST 2017

We also have 3 other switches that are on the same firmware that haven’t given those errors, but they’ve only been upgraded for 7 days.

pastebin output of ./poller.php -h HOSTNAME -d -m ports
and ./discovery.php -h HOSTNAME -d -m ports

Poller output: https://pastebin.com/XbtPTqSh
Discovery output: https://pastebin.com/Md1YsLxg