Strange things are happening in discovery_wrapper.log

Help
,
1

Hi everyone, I’ve noticed that lately my Libre was having some gaps in the graphs and very often the poller phase would time out >900 seconds. So I started reading some log files. poller-wrapper.log and discovery-wrapper.log below, I found this:

[VRF **nVSatellite] PORTS - PHP Error(2): Trying to access array offset on null in /opt/librenms/includes/discovery/vrf.inc.php:148
PHP Error(2): Trying to access array offset on null in /opt/librenms/includes/discovery/vrf.inc.php:149
PHP Error(2): Trying to access array offset on null in /opt/librenms/includes/discovery/vrf.inc.php:150

Now the polling is for a Cisco asr9k. The vrf **nvSatellite is actually a name chosen by Cisco. And I think this is causing the problem, namely the asterisk. I tried deleting it in the conf, but to no avail. Is it possible to hide it from librenms, knowing the OID of this “interface”? I confess I don’t know what to do.
Thanks in advance, as always!

2

This would not cause your polling to take a long time (it would make it faster if it caused an issue).

The error is because it could not find the port the vrf is connected to.

You need to look into your high polling times separately. I suggest the polling module graph to see which modules take a long time as a starting point.

3

Hi Murrant, thanks for your reply! I tried analyzing the log output and found some interesting things. For example, I saw that the problem occurs not only with the Cisco ASR9K family but also with the Cisco N540 family (also iOSXR). In practice I see this:

Load disk module route

IP-FORWARD-MIB::inetCidrRouteTable
IP-FORWARD-MIB::ipCidrRouteTable
IPV6-MIB::ipv6RouteTable
RFC1213-MIB::ipRouteTable
MPLS-L3VPN-STD-MIB

2025-11-17 09:14:11,182 :: ERROR :: Thread Thread-2 (poll_worker) exited with code -254
2025-11-17 09:14:11,182 :: ERROR :: Timeout of 900 seconds expired for command “/usr/bin/env php /opt/librenms/discovery.php -h 32” execution. Original output was: LibreNMS Discovery
X.X.X.X 32 iosxr

It basically seems to be related to this module, although I don’t know how. The machines are in a classic configuration, with the data stream managed with VRF and mpls ldp for the pseudowires and bgp vpnv4 for the transport.
I’d like to try disabling polling on routes, but only as a test!

Thanks in advance

1 Like
4

That output is from the route module, not the vrf module.
Likely, it is trying to walk a full routing table or something stupid. It is supposed to avoid that, but maybe it isn’t working in your case.

You should try pastebinning the output of lnms device:poll -vv -m route <device>

5

This output refers to one of the machines exhibiting this strange behavior. Now that I know this command, I’ll try to check the others as well.

Thanks

6

It’s discovery that is causing the issue so run ./discovery.php -h <device> ID -v -m route

7

Hi Iaf, thanks for the reply. I do notice that there are no errors in this case, but the discovery phase is still very slow.
So, to summarize, the problem occurs on both ASR9K family machines and the new N540s.

Load disco module route

IP-FORWARD-MIB::inetCidrRouteTable
IP-FORWARD-MIB::ipCidrRouteTable
IPV6-MIB::ipv6RouteTable
RFC1213-MIB::ipRouteTable
MPLS-L3VPN-STD-MIB

This single module takes over 800 seconds, but not always and not on all devices. I’m completely confused!
This is the pastebin output.
Thanks!
https://pastebin.com/L8iRit7u

8

pastebin link doesn’t work

9

https://pastebin.com/0y1z3AKL

actually the pastebin links are in pending moderation status

10

Hi everyone, Pastebin had actually blocked my uploaded files, but I didn’t understand why! Anyway, this file seems to be fine now.

https://pastebin.com/UJCa18zR

P.S

I noticed that when loading the route module, the MIB that causes problems is MPLS-L3VPN-STD-MIB. It wastes all 800 seconds there. Now I’m trying to test that MIB individually on some machines.

11

I still can’t get to those pastebin links. Maybe it’s just me though.

But yes, if you can see one snmp query is being blocked for 800 seconds then you need to test just that and understand why.

12

You’re right! My file (our old .txt file) is being put in pending moderation by Pastebin’s spam filter. I’m confused!
Anyway, I emailed [email protected] to try to find a solution together. I hope they’ll respond. The link to Pastebin is:
https://pastebin.com/UJCa18zR

13

Probably quicker to find another pastebin service.

14

I’m really sorry for this waste of time, but yes, you’re right. I relied on hastebin and it seems to work… I hope!

15

Re-run discovery.php with -d as well.

Watch out as it will possibly have some sensitive info so check it before putting it on a pastebin.

16

Hi Neil, some interesting things actually came out with the -d option.:+1:

17

You can do a couple of things for this:

  1. Just disable the route module for that device if you don’t need it.
  2. Restrict which OIDs are accepted on your device so less data is sent back.
18

I ran this test:
/usr/bin/snmpbulkwalk’ ‘-M’ ‘/opt/librenms/mibs:/opt/librenms/mibs/cisco’ ‘-m’ ‘SNMPv2-TC:SNMPv2-MIB:IF-MIB:IP-MIB:TCP-MIB:UDP-MIB:NET-SNMP-VACM-MIB’ ‘-v2c’ ‘-c’ ‘community’ ‘-OQXUte’ ‘-Pu’ ‘-Os’ ‘udp:x.y.n.z:161’ ‘MPLS-L3VPN-STD-MIB::mplsL3VpnVrfRteTable’

I found that it doesn’t throw any errors. The amount of data this MIB “pulls” is simply impressive. Very difficult to manage.

Considering the amount of data, I wonder if it’s possible to do a single discovery every 24 hours. Perhaps during nighttime. Do you think this is possible? These are backbone machines, so this data is important to us. The alternative is to explore the MPLS-L3VPN-STD-MIB and discover the oids related to the routes divided by VRF.

19

You can’t change the discovery time on a per device basis I’m afraid, it’s global only.

20

But thinking about it, I think it could work on a global basis too. Let me explain: I monitor about 300 machines, and if I could get them to run the discovery phases from 12:00 AM to 7:00 AM every day, that would be a great result for me. I saw that there’s a discovery-wrapper.py file… but I’d rather ask how. My level of expertise on this subject is low!
Thanks in advance