IOS-XE discovery bug? since 1.68 update

Hello

We had some odd new bug which i think is either discovery or polling related, since the update of 1.68 on some IOS-XE devices, validate and librenms.log output when the error starts below:

When the error starts in librenms.log file, all interfaces are set to NULL on the IOS-XE devices.

Eventlog snippet
image

Librenms.log snippet

/opt/librenms/poller.php 135 2020-10-01 00:15:13 - 1 devices polled in 150.2 secs
/opt/librenms/poller.php 160 2020-10-01 00:15:13 - 1 devices polled in 148.5 secs
/opt/librenms/poller.php 162 2020-10-01 00:15:14 - 1 devices polled in 149.1 secs
[2020-10-01 00:15:15] production.ERROR: Call to undefined method LibreNMS\OS\Iosxe::getDeviceArray() {"exception":"[object] (Error(code: 0): Call to undefined method LibreNMS\\OS\\Iosxe::getDeviceArray() at /opt/librenms/LibreNMS/Device/Sensor.php:310)
[stacktrace]
#0 /opt/librenms/includes/polling/wireless.inc.php(29): LibreNMS\\Device\\Sensor::poll(Object(LibreNMS\\OS\\Iosxe))
#1 /opt/librenms/includes/polling/functions.inc.php(315): include('/opt/l...')
#2 /opt/librenms/poller.php(142): poll_device(Array, false)
#3 {main}
"}
[2020-10-01 00:15:16] production.ERROR: Call to undefined method LibreNMS\OS\Iosxe::getDeviceArray() {"exception":"[object] (Error(code: 0): Call to undefined method LibreNMS\\OS\\Iosxe::getDeviceArray() at /opt/librenms/LibreNMS/Device/Sensor.php:310)
[stacktrace]
#0 /opt/librenms/includes/polling/wireless.inc.php(29): LibreNMS\\Device\\Sensor::poll(Object(LibreNMS\\OS\\Iosxe))
#1 /opt/librenms/includes/polling/functions.inc.php(315): include('/opt/l...')
#2 /opt/librenms/poller.php(142): poll_device(Array, false)
#3 {main}
"}
[2020-10-01 00:15:17] production.ERROR: Call to a member function getOriginal() on array {"exception":"[object] (Error(code: 0): Call to a member function getOriginal() on array at /opt/librenms/LibreNMS/Modules/OS.php:97)
[stacktrace]
#0 /opt/librenms/LibreNMS/Modules/OS.php(81): LibreNMS\\Modules\\OS->handleChanges(Object(LibreNMS\\OS\\Iosxe))
#1 /opt/librenms/includes/polling/os.inc.php(8): LibreNMS\\Modules\\OS->poll(Object(LibreNMS\\OS\\Iosxe))
#2 /opt/librenms/includes/polling/functions.inc.php(315): include('/opt/l...')
#3 /opt/librenms/poller.php(142): poll_device(Array, false)
#4 {main}
"}
[2020-10-01 00:15:17] production.ERROR: Call to a member function getOriginal() on array {"exception":"[object] (Error(code: 0): Call to a member function getOriginal() on array at /opt/librenms/LibreNMS/Modules/OS.php:97)
[stacktrace]
#0 /opt/librenms/LibreNMS/Modules/OS.php(81): LibreNMS\\Modules\\OS->handleChanges(Object(LibreNMS\\OS\\Iosxe))
#1 /opt/librenms/includes/polling/os.inc.php(8): LibreNMS\\Modules\\OS->poll(Object(LibreNMS\\OS\\Iosxe))
#2 /opt/librenms/includes/polling/functions.inc.php(315): include('/opt/l...')
#3 /opt/librenms/poller.php(142): poll_device(Array, false)

(Errors continue for 2 minutes, I can get full log if that will help)

After 5 minutes the interfaces return to normal state, so not sure what happened. Maybe related to the new discovery changes in 1.68?

Validate.php

====================================
Component | Version
--------- | -------
LibreNMS  | 1.68
DB Schema | 2020_09_22_172321_add_alert_log_index (174)
PHP       | 7.3.18
Python    | 3.6.8
MySQL     | 5.5.65-MariaDB
RRDTool   | 1.7.1
SNMP      | NET-SNMP 5.7.2
====================================

[OK]    Composer Version: 1.10.13
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct

Any ideas ?

Thanks

It seems like something could be wrong in your local install… what does git status show?

Thanks murrant,

I see now that I have an old pppoe custom discover for ios-xe so thanks for pointing me in the right direction.

I have this file in untracked “includes/discovery/os/iosxe.inc.php”, i realized this is due to custom discovery i setup ages ago for PPPoE Sessions for “CISCO-AAA-SESSION-MIB” for ios-xe devices ages ago Graphing PPPoE sessions on Cisco device

i’m guessing this now causes a problem with the new discovery changes maybe? i’ll delete the file and hopefully it is fixed now :slight_smile:

 git status
HEAD detached at 90d0d46df
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        deleted:    rrd/.gitignore


Untracked files:
  (use "git add <file>..." to include in what will be committed)

        html/includes/
        includes/discovery/os/iosxe.inc.php
        includes/html/print-menubar-custom.inc.php
        includes/testing.php
        mibs/CISCO-PPPOE-MIB
        rrd/x.x.x.x
        rrd/x.x.x.x
        rrd/x.x.x.x
        rrd/x.x.x.x
        rrd/x.x.x.x
        rrd/x.x.x.x
        rrd/x.x.x.x
        rrd/x.x.x.x
        rrd/x.x.x.x
        rrd/x.x.x.x
        rrd/x.x.x.x
        rrd/x.x.x.x
        rrd/x.x.x.x




Yeah, if you want to port it, I would move it to polling, the discovery/os/*.inc.php is meant for OS detection.

Probably the best thing would be to move it to a “count” sensor in Yaml.

1 Like

Great! thanks so much