Looking for testers for port polling update

laf · 7 February 2017 19:59

Thanks

Would one of you be able to get me the following info:

Polling time before the change
Polling time after the change.
The ‘time’ value of this command: time snmpbulkwalk -OUneb -m IF-MIB -M /opt/librenms/mibs -v2c -c COMMUNITY HOSTNAME ifOperStatus

twilley · 8 February 2017 00:45

Here are two gists (I wasn’t sure exactly which time values would be useful to you, so I used ./poller.php -h HOSTNAME | grep " seconds$\\|s\]$" to get all of the timing-related output):

A fresh standalone Juniper EX3300-48P (nothing active, nothing disabled)

gist.github.com

https://gist.github.com/twilley/1ad97746bf678bf96bdbdbd2b242b814

fresh-ex3300

Fresh Juniper EX3300-48P standalone switch with no active access ports, and no ports disabled

Before activating selected_ports attribute:
>> Runtime for poller module 'os': 0.0881 seconds
>> Runtime for poller module 'ipmi': 0.0006 seconds
>> Runtime for poller module 'sensors': 0.0615 seconds
>> Runtime for poller module 'processors': 0.0438 seconds
>> Runtime for poller module 'mempools': 0.2632 seconds
>> Runtime for poller module 'storage': 1.3438 seconds
>> Runtime for poller module 'netstats': 1.3782 seconds

This file has been truncated. show original

A production Juniper EX3300-48P virtual-chassis with 8 member switches (many active/inactive access ports, many deactivated ports)

gist.github.com

https://gist.github.com/twilley/80f0d06064d27977cc9abb922cc1ee0f

production-ex3300

Production Juniper EX3300-48P virtual-chassis switch with 8 member switches with a mix of active/inactive access ports and disabled ports

Before activating selected_ports attribute:
>> Runtime for poller module 'os': 0.0945 seconds
>> Runtime for poller module 'ipmi': 0.0006 seconds
>> Runtime for poller module 'sensors': 0.1835 seconds
>> Runtime for poller module 'processors': 0.2096 seconds
>> Runtime for poller module 'mempools': 0.7302 seconds
>> Runtime for poller module 'storage': 0.9224 seconds
>> Runtime for poller module 'netstats': 0.4095 seconds

This file has been truncated. show original

I hope that helps.

laf · 8 February 2017 08:41

Actually I was just after the output at the end but that is even better

laf · 8 February 2017 09:31

Ok so I’ve updated the code to now ignore ports which are currently down and were down on the last poll, if anyone wants to try that then just git pull on my branch

laf · 8 February 2017 10:19

Hang fire, 1 bug

laf · 8 February 2017 12:05

All updated again. I can’t compare before the check for ports down but I’ve seen a few seconds of a couple of devices which have about 1300 ports, non-deleted but a lot of them are down.

wriedel · 9 February 2017 11:26

Hi Laf,
any intentions to make this also work with Python 3?
I already “hacked” the existing script to make it work and I am more then happy to hand it over into professional hands for wider consumption. Just let me know.

laf · 9 February 2017 11:50

Wrong topic?

twilley · 13 February 2017 00:55

Ok, after running this for a while, I think there may be a need (for me) to better understand what this is doing. Some observations and possible issues:

ports that are disabled in the switch configuration don’t show up in the Overview and Ports tabs. This isn’t really a problem for me, just and observation that may be confusing.
ports that are disabled for a time, and then are reactivated aren’t polled until after the switch gets re-discovered.
some weirdness with Juniper where an active logical interface, say ge-0/0/46.0, is polled, but the parent physical interface, ge-0/0/46, is marked as deleted and is not polled… Not sure how this gets messed up, nor how to fix.

I’d have to spend some more time learning what this code does in order to even begin to recommend specific changes, but here are my thoughts:

What are the criteria for selecting ports to not poll? I’m guessing checking whether ifAdminStatus == down. Is that correct?
To flag the ports that we don’t want to waste time polling, I’m guessing the mechanism is to set deleted = 1. Is that correct?
Regardless of the flag, I would say that each polling cycle we should check the ifAdminStatus of each port (poll the switch for this value, not looking at the DB)
- if (ifAdminStatus != down and deleted != 0) then set deleted = 0
- if (ifAdminStatus == down and ifAdminStatus_prev == down and deleted != 1) then set deleted = 1
After step 2 above, do the detailed interface poll on the ports where deleted != 1

laf · 13 February 2017 09:33

From the code, the only way a port will be deleted is if it doesn’t pass the is_port_valid() check which would happen without this change in code from what I can see.

Ideally I’d need to see the output of ./poller.php -h HOSTNAME -r -f -d -m ports

twilley · 14 February 2017 09:20

Ok, I hacked together some code that seems to do what I think it should be doing. Replace your includes/ports.inc.php file with the contents of this gist: https://gist.github.com/twilley/117043c1f2b1112469bf8e9325135d79 a diff can be seen here: https://gist.github.com/twilley/8af93d10a0c924691ff3063f4d8bf285

my changes are:

add a block of code where we check to see the admin and oper status of the deleted ports, if they are up/up, we update the database and set deleted = 0 for these ports. this happens before we loop through the nondeleted ports
instead of checking ifOperStatus, ifOperStatus_prev, ifAdminStatus, and ifAdminStatus_prev to see if they’re “down”, check to see if they’re “not up” - this is important at least for ifOperStatus, as the value can be up, down, or lowerLayerDown (maybe other values too?)
later on, there is some code that would only update the ifOperStatus_prev and ifAdminStatus_prev values in the database if the value is null. I was finding that the values would sometimes not be updated at all, so removed the conditional.
I added/modified a some echo statements that could be cut out before merging.

wriedel · 27 February 2017 11:46

Hi Folks, I think it would be beneficial to have some tweaking especially if we’re heading for 1 minute fast polling. I guess it would make sense to have something like enable modules per device group or bake more intelligence into the polling. Why even try polling for OTV on an ASA or a Linux server or IPMI on a virtual machine?
Also I just wonder if it’s feasible to dynamically group devices or modules by OID response time. Like while I would like to poll IPSLA every minute don’t do that for my full BGP table.

laf · 28 February 2017 12:44

Further enhancement to this feature if anyone wants to test:

https://github.com/librenms/librenms/pull/6037