Inccorrect values logged for Cisco Catalyst C9500 switch

Help
1

Hi,

I just discovered that some of the sensors on our Cisco C9500 switch are logged incorectly. Thay seem to be off by a factor of 10.
When the SNMP value is 26, what Libre logs is 2.6. This is specifically the optical transivers:

/usr/bin/snmpbulkwalk -v2c -c PUBLIC -OQUs -m ENTITY-MIB -M /opt/librenms/mibs:/opt/librenms/mibs/cisco udp:127.0.0.1:161 entPhysicalName
entPhysicalName.1174 = TwentyFiveGigE1/0/15
entPhysicalName.1175 = Twe1/0/15 Module Temperature Sensor
entPhysicalName.1176 = Twe1/0/15 Supply Voltage Sensor
entPhysicalName.1177 = Twe1/0/15 Bias Current Sensor
entPhysicalName.1178 = Twe1/0/15 Transmit Power Sensor
entPhysicalName.1179 = Twe1/0/15 Receive Power Sensor

/usr/bin/snmpget -v2c -c PUBLIC -OUQnte -M /opt/librenms/mibs:/opt/librenms/mibs/cisco udp:127.0.0.1:161 .1.3.6.1.4.1.9.9.91.1.1.1.1.4.<Above IDs>
.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1175 = 26
.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1176 = 3
.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1177 = 18
.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1178 = -6
.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1179 = -25

These values corespond to what the switch is telling me on the CLI:

Switch1#sho int twe1/0/15 transceiver detail

ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, +  : high warning, -  : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.

                                High Alarm  High Warn  Low Warn   Low Alarm
             Temperature        Threshold   Threshold  Threshold  Threshold
Port         (Celsius)          (Celsius)   (Celsius)  (Celsius)  (Celsius)
---------    -----------------  ----------  ---------  ---------  ---------
Twe1/0/15    26.8                  110.0       95.0      -42.0      -45.0

                                High Alarm  High Warn  Low Warn   Low Alarm
             Voltage            Threshold   Threshold  Threshold  Threshold
Port         (Volts)            (Volts)     (Volts)    (Volts)    (Volts)
---------    -----------------  ----------  ---------  ---------  ---------
Twe1/0/15    3.30                   3.60       3.50       3.05       3.00

                                High Alarm  High Warn  Low Warn   Low Alarm
             Current            Threshold   Threshold  Threshold  Threshold
Port         (milliamperes)     (mA)        (mA)       (mA)       (mA)
---------    -----------------  ----------  ---------  ---------  ---------
Twe1/0/15    18.4                   80.0       70.0        3.0        2.0

             Optical            High Alarm  High Warn  Low Warn   Low Alarm
             Transmit Power     Threshold   Threshold  Threshold  Threshold
Port         (dBm)              (dBm)       (dBm)      (dBm)      (dBm)
---------    -----------------  ----------  ---------  ---------  ---------
Twe1/0/15    -6.3                   -1.0       -2.0       -9.0      -10.0

             Optical            High Alarm  High Warn  Low Warn   Low Alarm
             Receive Power      Threshold   Threshold  Threshold  Threshold
Port         (dBm)              (dBm)       (dBm)      (dBm)      (dBm)
---------    -----------------  ----------  ---------  ---------  ---------
Twe1/0/15   -25.5                   -2.0       -3.0      -22.0      -23.0

LibreNMS manual polling:

sudo -u librenms ./poller.php -h 127.0.0.1 -d -m sensors -v

SNMP['/usr/bin/snmpget' '-v2c' '-c' 'PUBLIC' '-OUQnte' '-M' '/opt/librenms/mibs:/opt/librenms/mibs/cisco' 'udp:127.0.0.1:161' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1133' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1139' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1145' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1151' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1157' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1163' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1169' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1175' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1217' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.2097']
.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1175 = 26


2.6 
RRD[last 127.0.0.1/sensor-temperature-cisco-entity-sensor-1175.rrd  --daemon LibreNMS.local.net:42217]
RRDtool Output: 1581245400
OK u:0.01 s:0.01 r:1.39
RRD[update 127.0.0.1/sensor-temperature-cisco-entity-sensor-1175.rrd N:2.6 --daemon LibreNMS.local.net:42217]
RRDtool Output: OK u:0.01 s:0.00 r:1.39

SNMP['/usr/bin/snmpget' '-v2c' '-c' 'SK2005IT' '-OUQnte' '-M' '/opt/librenms/mibs:/opt/librenms/mibs/cisco' 'udp:127.0.0.1:161' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1134' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1140' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1146' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1152' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1158' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1164' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1170' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1176' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1218' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.2098']
.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1176 = 3

0.3 
RRD[last 127.0.0.1/sensor-voltage-cisco-entity-sensor-1176.rrd  --daemon LibreNMS.local.net:42217]
RRDtool Output: 1581245400
OK u:0.01 s:0.01 r:1.68
RRD[update 127.0.0.1/sensor-voltage-cisco-entity-sensor-1176.rrd N:0.3 --daemon LibreNMS.local.net:42217]
RRDtool Output: OK u:0.01 s:0.00 r:1.68

SNMP['/usr/bin/snmpget' '-v2c' '-c' 'SK2005IT' '-OUQnte' '-M' '/opt/librenms/mibs:/opt/librenms/mibs/cisco' 'udp:127.0.0.1:161' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1135' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1141' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1147' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1153' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1159' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1165' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1171' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1177' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1219' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.2099']
.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1177 = 18

0.0018 
RRD[last 127.0.0.1/sensor-current-cisco-entity-sensor-1177.rrd  --daemon LibreNMS.local.net:42217]
RRDtool Output: 1581245400
OK u:0.00 s:0.00 r:0.54
RRD[update 127.0.0.1/sensor-current-cisco-entity-sensor-1177.rrd N:0.0018 --daemon LibreNMS.local.net:42217]
RRDtool Output: OK u:0.00 s:0.00 r:0.54

SNMP['/usr/bin/snmpget' '-v2c' '-c' 'SK2005IT' '-OUQnte' '-M' '/opt/librenms/mibs:/opt/librenms/mibs/cisco' 'udp:127.0.0.1:161' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1166' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1167' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1172' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1173' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1178' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1179' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1220' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1221' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.2100' '.1.3.6.1.4.1.9.9.91.1.1.1.1.4.2101']
.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1178 = -6
.1.3.6.1.4.1.9.9.91.1.1.1.1.4.1179 = -25

-0.6 
RRD[last 127.0.0.1/sensor-dbm-cisco-entity-sensor-1178.rrd  --daemon LibreNMS.local.net:42217]
RRDtool Output: 1581245400
OK u:0.01 s:0.00 r:1.00
RRD[update 127.0.0.1/sensor-dbm-cisco-entity-sensor-1178.rrd N:-0.6 --daemon LibreNMS.local.net:42217]
RRDtool Output: OK u:0.01 s:0.00 r:1.00

-2.5 
RRD[last 127.0.0.1/sensor-dbm-cisco-entity-sensor-1179.rrd  --daemon LibreNMS.local.net:42217]
RRDtool Output: 1581245400
OK u:0.01 s:0.00 r:1.01
RRD[update 127.0.0.1/sensor-dbm-cisco-entity-sensor-1179.rrd N:-2.5 --daemon LibreNMS.local.net:42217]
RRDtool Output: OK u:0.01 s:0.00 r:1.00

LibreNMS ./validate.php

====================================
Component | Version
--------- | -------
LibreNMS  | 1.60-9-gf8ce4df0f
DB Schema | 2020_02_05_224042_device_inserted_null (159)
PHP       | 7.2.24-0ubuntu0.18.04.2
MySQL     | 10.1.44-MariaDB-0ubuntu0.18.04.1
RRDTool   | 1.7.0
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.9.3
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
2

When you manually poll with -d option, search for the output of entSensorScale as LibreNMS calculate it from that.

3

When running ./poller.php -h 127.0.0.1 -d I can’t find the string entSensorScale in any of the output.

4

Sorry, I was suposed to say discovery, not poller. My fault.

5

So from what I can see in the discovery it seems to discover correct.
I am not quite sure what the entSensorPrecision does to the values, but the entSensorValue are the correct values. But when the poller logs the information they are reduced by a factor of 10.

Snippets from ./discovery.php -h 127.0.0.1 -d -v:

 entSensorScaleSQL[select `hostname`, `overwrite_ip` from `devices` where `hostname` = ? limit 1 ["127.0.0.1"] 0.47ms] 
  
SNMP['/usr/bin/snmpbulkwalk' '-v2c' '-c' 'PUBLIC' '-OQUs' '-m' 'CISCO-ENTITY-SENSOR-MIB' '-M' '/opt/librenms/mibs:/opt/librenms/mibs/cisco' 'udp:127.0.0.1:161' 'entSensorScale']


entSensorScale.1175 = units
entSensorScale.1176 = units
entSensorScale.1177 = milli
entSensorScale.1178 = units
entSensorScale.1179 = units


 entSensorValueSQL[select `hostname`, `overwrite_ip` from `devices` where `hostname` = ? limit 1 ["127.0.0.1"] 0.42ms] 
  
SNMP['/usr/bin/snmpbulkwalk' '-v2c' '-c' 'PUBLIC' '-OQUs' '-m' 'CISCO-ENTITY-SENSOR-MIB' '-M' '/opt/librenms/mibs:/opt/librenms/mibs/cisco' 'udp:127.0.0.1:161' 'entSensorValue']

entSensorValue.1175 = 26
entSensorValue.1176 = 3
entSensorValue.1177 = 18
entSensorValue.1178 = -6
entSensorValue.1179 = -25

 entSensorPrecisionSQL[select `hostname`, `overwrite_ip` from `devices` where `hostname` = ? limit 1 ["127.0.0.1"] 0.93ms] 
  
SNMP['/usr/bin/snmpbulkwalk' '-v2c' '-c' 'PUBLIC' '-OQUs' '-m' 'CISCO-ENTITY-SENSOR-MIB' '-M' '/opt/librenms/mibs:/opt/librenms/mibs/cisco' 'udp:127.0.0.1:161' 'entSensorPrecision']

entSensorPrecision.1175 = 1
entSensorPrecision.1176 = 1
entSensorPrecision.1177 = 1
entSensorPrecision.1178 = 1
entSensorPrecision.1179 = 1

array (
  1175 => 
  array (
    'entSensorType' => 'celsius',
    'entSensorScale' => 'units',
    'entSensorValue' => '26',
    'entSensorMeasuredEntity' => '1174',
    'entSensorPrecision' => '1',
  ),
  1176 => 
  array (
    'entSensorType' => 'voltsDC',
    'entSensorScale' => 'units',
    'entSensorValue' => '3',
    'entSensorMeasuredEntity' => '1174',
    'entSensorPrecision' => '1',
  ),
  1177 => 
  array (
    'entSensorType' => 'amperes',
    'entSensorScale' => 'milli',
    'entSensorValue' => '18',
    'entSensorMeasuredEntity' => '1174',
    'entSensorPrecision' => '1',
  ),
  1178 => 
  array (
    'entSensorType' => 'dBm',
    'entSensorScale' => 'units',
    'entSensorValue' => '-6',
    'entSensorMeasuredEntity' => '1174',
    'entSensorPrecision' => '1',
  ),
  1179 => 
  array (
    'entSensorType' => 'dBm',
    'entSensorScale' => 'units',
    'entSensorValue' => '-25',
    'entSensorMeasuredEntity' => '1174',
    'entSensorPrecision' => '1',
  ),
)
6

The entSensorPrecision is for calculating exactly that, as you can see here https://github.com/librenms/librenms/blob/master/includes/discovery/sensors/cisco-entity-sensor.inc.php

 if ($entry['entSensorScale'] == 'nano') {
                    $divisor    = '1000000000';
                    $multiplier = '1';
                }

                if ($entry['entSensorScale'] == 'micro') {
                    $divisor    = '1000000';
                    $multiplier = '1';
                }

                if ($entry['entSensorScale'] == 'milli') {
                    $divisor    = '1000';
                    $multiplier = '1';
                }

                if ($entry['entSensorScale'] == 'units') {
                    $divisor    = '1';
                    $multiplier = '1';
                }

                if ($entry['entSensorScale'] == 'kilo') {
                    $divisor    = '1';
                    $multiplier = '1000';
                }

                if ($entry['entSensorScale'] == 'mega') {
                    $divisor    = '1';
                    $multiplier = '1000000';
                }

                if ($entry['entSensorScale'] == 'giga') {
                    $divisor    = '1';
                    $multiplier = '1000000000';
                }

So that factor of 10 reduction should come from another place of the code

7

Just below the code you pasted, we have the following:

if (is_numeric($entry['entSensorPrecision']) && $entry['entSensorPrecision'] > '0') {
  $divisor = $divisor.str_pad('', $entry['entSensorPrecision'], '0');
}

$current = ($current * $multiplier / $divisor);

If i read that correctly would that not turn entSensorPrecision.1179 = 1 into a value of 10?

8

I found the following post on Cisco’s community:

Snippet from that:
Now for example we have to interpret values for transmit and receive power of TE2/4. So

SNMPv2-SMI::enterprises.9.9.91.1.1.1.1.1.2061 = INTEGER: 14  -----> means dBm
SNMPv2-SMI::enterprises.9.9.91.1.1.1.1.2.2061 = INTEGER: 9   -----> means units
SNMPv2-SMI::enterprises.9.9.91.1.1.1.1.3.2061 = INTEGER: 1   -----> number of decimal place is one
SNMPv2-SMI::enterprises.9.9.91.1.1.1.1.4.2061 = INTEGER: -19 -----> after placing decimal after one integer value is -1.9 dBm which is actual value

So maybe this is a bug in the Cisco firmware on our C9500 Switches?
As LibreNMS does exactly as described in the post.

Are here others that see this issue, the switch in question is:
C9500-24Y4C - Cisco IOS-XE CAT9K_IOSXE 16.9.3 (Fuji)

9

Same problem, different device, different position in the code: https://community.librenms.org/t/megatec-netagent-ii-adpos-psu-internal-battery-voltage-wrong

Here also a division by 10 is performed, but not necessary - only, the OID delivers the wrong value, as the device doesn’t behave in conformity to the RFC.

10

I just checked a different type switch in the same model, and this one is reporting correctly in LibreNMS.
It’s entSensorValue is a factor of 10 higher than what i get from the other switch type.

The working switch is a C9500-16X - Cisco IOS-XE CAT9K_IOSXE 16.9.3 (Fuji).

So this seems to be a problem with this particular switch type: C9500-24Y4C - Cisco IOS-XE CAT9K_IOSXE 16.9.3 (Fuji).

11

Cisco TAC has confirmed this is a bug in this particular switch model (C9500-24Y4C).

Wrong values for transceivers (DOM) in C9500-24Y4C and C9500-48Y4C
bug CSCvt16172