Broken VDSL stats after update

Tags: #<Tag:0x00007f6cb627d600> #<Tag:0x00007f6cb627d538>

G’day.

I had to do a manual update as daily.sh didn’t do anything.
(I did this before on another install, and didn’t have any issues…)

This update has killed some graphing on a Cisco VDSL router. (other graphs like traffic are working fine.) but now it’s not showing line stats (snr etc.) on ATM0-adsl - ATM0 is not used, but still used to graph the snr etc.
the router is an 887VA / 15.4(3)M10
I’ve attemtped to rediscover but. still the same… I don’t relally want to nuke the device as there’s a couple of years of history.

The install was about six months old ./validate shows all ok, so any db updates I presume it’s done…

Any thoughts about where to start looking as librenms.log or error.log isn’t showing anything about unable to poll correctly ?

TIA

MarkA

I think some rrd broken yes

if u debug the polling u will see:
RRD warning: unused data sent adslAtucCurrSnrMgn
RRD warning: unused data sent adslAtucCurrAtn
RRD warning: unused data sent adslAtucCurrOutputPwr
RRD warning: unused data sent adslAtucCurrAttainableRate
RRD warning: unused data sent adslAtucChanCurrTxRate
RRD warning: unused data sent adslAturCurrSnrMgn
RRD warning: unused data sent adslAturCurrAtn
RRD warning: unused data sent adslAturCurrOutputPwr
RRD warning: unused data sent adslAturCurrAttainableRate
RRD warning: unused data sent adslAturChanCurrTxRate
RRD warning: unused data sent adslAtucPerfLofs
RRD warning: unused data sent adslAtucPerfLoss
RRD warning: unused data sent adslAtucPerfLprs
RRD warning: unused data sent adslAtucPerfESs
RRD warning: unused data sent adslAtucPerfInits
RRD warning: unused data sent adslAturPerfLofs
RRD warning: unused data sent adslAturPerfLoss
RRD warning: unused data sent adslAturPerfLprs
RRD warning: unused data sent adslAturPerfESs
RRD warning: unused data sent adslAtucChanCorrectedBlks
RRD warning: unused data sent adslAtucChanUncorrectBlks
RRD warning: unused data sent adslAturChanCorrectedBlks
RRD warning: unused data sent adslAturChanUncorrectBlks

so they need to strip the adsl i think bc rrds are whitout adsl tag

Hmm, this may have something to do with it !

From the poller log.
ATM0-adsl
346 skipped because port is still down since last polling.

and that’s where the VDSL line data is…(was.)

Is there a way to force polling a perticular port even if it’s reporting down for admin/phy ?

No, @SniperVegeta has the correct answer. I’ll merge something.

Sadly, still not working. What sort of debug is needed ?

Here its working again…

ok, plouging through logs, It looks the snmpwalk takes place, reads, and then places it in to array… then it dosen’t get inserted in sql… ignored.
If I’m looking in the wrong place, plese let me know…
p.s. I’m not dev, just do systems stuff…

ADSL
SNMP[’/usr/bin/snmpbulkwalk’ ‘-v2c’ ‘-c’ ‘COMMUNITY’ ‘-OQUs’ ‘-m’ ‘ADSL-LINE-MIB’ ‘-M’ ‘/opt/librenms/mibs:/opt/librenms/mibs/cisco’ ‘udp:HOSTNAME:161’ ‘.1.3.6.1.2.1.10.94.1’]

adslLineCoding.14 = dmt
adslLineType.14 = fastOnly
adslAtucInvVendorID.14 = B500BDCM
adslAtucInvVersionNumber.14 = 0xa4a1
adslAtucCurrSnrMgn.14 = 208
adslAtucCurrAtn.14 = 0
adslAtucCurrOutputPwr.14 = 44
adslAtucCurrAttainableRate.14 = 30425000
adslAturInvSerialNumber.14 = FCZ1747C2DW C887VA-W 15.4(3)M10
adslAturInvVendorID.14 = B500BDCM
adslAturInvVersionNumber.14 = 15.4(3)M10
adslAturCurrSnrMgn.14 = 176
adslAturCurrAtn.14 = 0
adslAturCurrOutputPwr.14 = 135
adslAturCurrAttainableRate.14 = 77772000
adslAtucChanCurrTxRate.14 = 39999000
adslAturChanCurrTxRate.14 = 9999000
adslAtucPerfLofs.14 = 0
adslAtucPerfLoss.14 = 0
adslAtucPerfLols.14 = 0
adslAtucPerfLprs.14 = 0
adslAtucPerfESs.14 = 1281
adslAtucPerfInits.14 = 7
adslAtucPerfCurr15MinLofs.14 = 0
adslAtucPerfCurr15MinLoss.14 = 0
adslAtucPerfCurr15MinLprs.14 = 0
adslAtucPerfCurr15MinESs.14 = 0
adslAtucPerfCurr15MinInits.14 = 7
adslAtucPerfCurr1DayLofs.14 = 0
adslAtucPerfCurr1DayLoss.14 = 0
adslAtucPerfCurr1DayLprs.14 = 0
adslAtucPerfCurr1DayESs.14 = 10
adslAtucPerfCurr1DayInits.14 = 7
adslAturPerfLofs.14 = 25
adslAturPerfLoss.14 = 26
adslAturPerfLprs.14 = 0
adslAturPerfESs.14 = 28
adslAturPerfCurr15MinLofs.14 = 0
adslAturPerfCurr15MinLoss.14 = 0
adslAturPerfCurr15MinLprs.14 = 0
adslAturPerfCurr15MinESs.14 = 0
adslAturPerfCurr1DayLofs.14 = 0
adslAturPerfCurr1DayLoss.14 = 0
adslAturPerfCurr1DayLprs.14 = 0
adslAturPerfCurr1DayESs.14 = 0
adslAtucChanCorrectedBlks.14 = 45723
adslAtucChanUncorrectBlks.14 = 2
adslAtucChanPerfCurr15MinCorrectedBlks.14 = 0
adslAtucChanPerfCurr15MinUncorrectBlks.14 = 0
adslAtucChanPerfCurr1DayCorrectedBlks.14 = 11
adslAtucChanPerfCurr1DayUncorrectBlks.14 = 0
adslAtucChanPerfPrev1DayCorrectedBlks.14 = 11
adslAtucChanPerfPrev1DayUncorrectBlks.14 = 0
adslAturChanCorrectedBlks.14 = 45723
adslAturChanUncorrectBlks.14 = 2
adslAturChanPerfCurr15MinCorrectedBlks.14 = 0
adslAturChanPerfCurr15MinUncorrectBlks.14 = 0
adslAturChanPerfCurr1DayCorrectedBlks.14 = 11
adslAturChanPerfCurr1DayUncorrectBlks.14 = 0
adslAtucChanIntervalCorrectedBlks.14.1 = 0
adslAtucChanIntervalCorrectedBlks.14.2 = 0
adslAtucChanIntervalCorrectedBlks.14.3 = 0
adslAtucChanIntervalCorrectedBlks.14.4 = 0
adslAtucChanIntervalUncorrectBlks.14.1 = 0
adslAtucChanIntervalUncorrectBlks.14.2 = 0
adslAtucChanIntervalUncorrectBlks.14.3 = 0
adslAtucChanIntervalUncorrectBlks.14.4 = 0
adslAtucChanIntervalValidData.14.1 = true
adslAtucChanIntervalValidData.14.2 = true
adslAtucChanIntervalValidData.14.3 = true
adslAtucChanIntervalValidData.14.4 = true
adslAturChanIntervalCorrectedBlks.14.1 = 0
adslAturChanIntervalCorrectedBlks.14.2 = 0
adslAturChanIntervalCorrectedBlks.14.3 = 0
adslAturChanIntervalCorrectedBlks.14.4 = 0
adslAturChanIntervalUncorrectBlks.14.1 = 0
adslAturChanIntervalUncorrectBlks.14.2 = 0
adslAturChanIntervalUncorrectBlks.14.3 = 0
adslAturChanIntervalUncorrectBlks.14.4 = 0
adslAturChanIntervalValidData.14.1 = true
adslAturChanIntervalValidData.14.2 = true
adslAturChanIntervalValidData.14.3 = true
adslAturChanIntervalValidData.14.4 = true

I then see it presumably inserted into an array

14 =>
array (
‘ifName’ => ‘AT0-adsl’,
‘ifLinkUpDownTrapEnable’ => ‘enabled’,
‘ifHighSpeed’ => ‘0’,
‘ifPromiscuousMode’ => ‘false’,
‘ifConnectorPresent’ => ‘false’,
‘ifAlias’ => ‘’,
‘ifCounterDiscontinuityTime’ => ‘0:0:00:24.82’,
‘ifDescr’ => ‘ATM0-adsl’,
‘ifAdminStatus’ => ‘up’,
‘ifOperStatus’ => ‘down’,
‘ifLastChange’ => ‘2761042533’,
‘ifType’ => ‘adsl’,
‘ifPhysAddress’ => ‘’,
‘ifMtu’ => ‘1600’,
‘adslLineCoding’ => ‘dmt’,
‘adslLineType’ => ‘fastOnly’,
‘adslAtucInvVendorID’ => ‘B500BDCM’,
‘adslAtucInvVersionNumber’ => ‘0xa4a1’,
‘adslAtucCurrSnrMgn’ => ‘208’,
‘adslAtucCurrAtn’ => ‘0’,
‘adslAtucCurrOutputPwr’ => ‘44’,
‘adslAtucCurrAttainableRate’ => ‘30425000’,
‘adslAturInvSerialNumber’ => ‘FCZ1747C2DW C887VA-W 15.4(3)M10’,
‘adslAturInvVendorID’ => ‘B500BDCM’,
‘adslAturInvVersionNumber’ => ‘15.4(3)M10’,
‘adslAturCurrSnrMgn’ => ‘176’,
‘adslAturCurrAtn’ => ‘0’,
‘adslAturCurrOutputPwr’ => ‘135’,
‘adslAturCurrAttainableRate’ => ‘77772000’,
‘adslAtucChanCurrTxRate’ => ‘39999000’,
‘adslAturChanCurrTxRate’ => ‘9999000’,
‘adslAtucPerfLofs’ => ‘0’,
‘adslAtucPerfLoss’ => ‘0’,
‘adslAtucPerfLols’ => ‘0’,
‘adslAtucPerfLprs’ => ‘0’,
‘adslAtucPerfESs’ => ‘1281’,
‘adslAtucPerfInits’ => ‘7’,
‘adslAtucPerfCurr15MinLofs’ => ‘0’,
‘adslAtucPerfCurr15MinLoss’ => ‘0’,
‘adslAtucPerfCurr15MinLprs’ => ‘0’,
‘adslAtucPerfCurr15MinESs’ => ‘0’,
‘adslAtucPerfCurr15MinInits’ => ‘7’,
‘adslAtucPerfCurr1DayLofs’ => ‘0’,
‘adslAtucPerfCurr1DayLoss’ => ‘0’,
‘adslAtucPerfCurr1DayLprs’ => ‘0’,
‘adslAtucPerfCurr1DayESs’ => ‘10’,
‘adslAtucPerfCurr1DayInits’ => ‘7’,
‘adslAturPerfLofs’ => ‘25’,
‘adslAturPerfLoss’ => ‘26’,
‘adslAturPerfLprs’ => ‘0’,
‘adslAturPerfESs’ => ‘28’,
‘adslAturPerfCurr15MinLofs’ => ‘0’,
‘adslAturPerfCurr15MinLoss’ => ‘0’,
‘adslAturPerfCurr15MinLprs’ => ‘0’,
‘adslAturPerfCurr15MinESs’ => ‘0’,
‘adslAtur
PerfCurr1DayLofs’ => ‘0’,
‘adslAturPerfCurr1DayLoss’ => ‘0’,
‘adslAturPerfCurr1DayLprs’ => ‘0’,
‘adslAturPerfCurr1DayESs’ => ‘0’,
‘adslAtucChanCorrectedBlks’ => ‘45723’,
‘adslAtucChanUncorrectBlks’ => ‘2’,
‘adslAtucChanPerfCurr15MinCorrectedBlks’ => ‘0’,
‘adslAtucChanPerfCurr15MinUncorrectBlks’ => ‘0’,
‘adslAtucChanPerfCurr1DayCorrectedBlks’ => ‘11’,
‘adslAtucChanPerfCurr1DayUncorrectBlks’ => ‘0’,
‘adslAtucChanPerfPrev1DayCorrectedBlks’ => ‘11’,
‘adslAtucChanPerfPrev1DayUncorrectBlks’ => ‘0’,
‘adslAturChanCorrectedBlks’ => ‘45723’,
‘adslAturChanUncorrectBlks’ => ‘2’,
‘adslAturChanPerfCurr15MinCorrectedBlks’ => ‘0’,
‘adslAturChanPerfCurr15MinUncorrectBlks’ => ‘0’,
‘adslAturChanPerfCurr1DayCorrectedBlks’ => ‘11’,
‘adslAturChanPerfCurr1DayUncorrectBlks’ => ‘0’,
),

then…

valid

valid

valid

valid

valid

valid

valid

valid

valid

ignored by ifDescr: ATM0-atm layer (matched: -atm)

SQL[UPDATE ports set deleted=? WHERE port_id = ? [“1”,null] 0.48ms]

ignored by ifDescr: ATM0.0-atm subif (matched: -atm)

ignored by ifType: aal5 (matched: aal5 )

ignored by ifType: aal5 (matched: aal5 )

valid

valid

valid

valid

valid

valid

valid

valid

ignored by ifDescr: Virtual-Access1 (matched: virtual-)

ignored by ifDescr: Virtual-Access2 (matched: virtual-)

ignored by ifDescr: VoIP-Null0 (matched: voip-null)

valid

valid

ignored by ifDescr: Virtual-Access3 (matched: virtual-)

ignored: empty ifDescr, ifAlias and ifName

ignored: empty ifDescr, ifAlias and ifName

ignored: empty ifDescr, ifAlias and ifName

ignored: empty ifDescr, ifAlias and ifName

So presume it’s gone somwhere ???

then I see this…

Port AT0-adsl: ATM0-adsl (14 / #346)
ifInOctets

ifOutOctets

ifInUcastPkts

ifOutUcastPkts

ifInBroadcastPkts

ifOutBroadcastPkts

ifInMulticastPkts

ifOutMulticastPkts

ifHighSpeed

VLAN =
Using ifDescr as ifAlias

ATM0-adsl
346 skipped because port is still down since last polling.

SQL[UPDATE ports set ifSpeed_prev=?,ifHighSpeed_prev=?,poll_time=?,poll_prev=?,poll_period=? WHERE port_id = ? [0,“0”,1586217524,1586217308,216,346] 1.72ms]

1 updated

and looking in the SQL table, the last update I can see on that interface is the day I upgrade
The RRD assocaited with the port ID, hasn’t been updated sine the “upgrade”

I still suspect it won’t read the port because it’s ‘down’. I agree why should it but for some reason, Cisco put live vaules there when it’s set for VDSL and the adsl int is not in use ???

Here is a temporary fix for anyone hitting this:

diff --git a/includes/polling/ports.inc.php b/includes/polling/ports.inc.php
index 0eb589b..71c7a5e 100644
--- a/includes/polling/ports.inc.php
+++ b/includes/polling/ports.inc.php
@@ -734,7 +734,7 @@ foreach ($ports as $port) {
         if (!empty($port['skipped'])) {
             // We don't care about statistics for skipped selective polling ports
             d_echo("$port_id skipped because selective polling ports is set.");
-        } elseif ($port['ifOperStatus'] == "down" && $port['ifOperStatus_prev'] == "down" && $this_port['ifOperStatus'] == "down" && $this_port['ifLastChange'] == $port['ifLastChange']) {
+        } elseif ($port['ifOperStatus'] == "down" && $port['ifOperStatus_prev'] == "down" && $this_port['ifOperStatus'] == "down" && $this_port['ifLastChange'] == $port['ifLastChange'] && $port['ifType'] != "adsl") {
             // We don't care about statistics for down ports on which states did not change since last polling
             d_echo("$port_id skipped because port is still down since last polling.");
         } else {

Obviously adding VDSL2 support is the way to properly fix this.

1 Like

Thank you!!!
This needs to be merged into production IMO

Looks like someone already has, https://github.com/librenms/librenms/commit/9797be367736cb284b43148e67741ff6b8a54440.

For ADSL, yes.
Not VDSL.

I was just looking for this solution, thanks!

@Dave
LibreNMS does not poll any vdsl mibs. Cisco is putting the xdsl statistics alltogether in the adsl mib, even if you have a vdsl line and modem. And because of this, the adsl interface is down (but the stats are still updated).
The fix allows the data to be polled even if the adsl interface is down.

On a Draytek router, the interface is of type “vdsl” on LibreNMS
I had to employ the same above logic through replace “adsl” with “vdsl” to achieve the same outcome of having LibreNMS poll the /down/ VDSL port for attentuation/sync speed etc

I just added my Draytek Vigor 130 to librenms, I hit the same problem. Port was discovered, but not polled. This fixed this:

index 2c9b6ddd5..f3371c9b7 100644
--- a/includes/polling/ports.inc.php
+++ b/includes/polling/ports.inc.php
@@ -735,7 +735,7 @@ foreach ($ports as $port) {
         if (! empty($port['skipped'])) {
             // We don't care about statistics for skipped selective polling ports
             d_echo("$port_id skipped because selective polling ports is set.");
-        } elseif ($port['ifType'] != 'adsl' && $port['ifOperStatus'] == 'down' && $port['ifOperStatus_prev'] == 'down' && $this_port['ifOperStatus'] == 'down' && $this_port['ifLastChange'] == $port['ifLastChange']) {
+        } elseif ($port['ifType'] != 'adsl' && $port['ifType'] != 'vdsl' && $port['ifOperStatus'] == 'down' && $port['ifOperStatus_prev'] == 'down' && $this_port['ifOperStatus'] == 'down' && $this_port['ifLastChange'] == $port['ifLastChange']) {
             // We don't care about statistics for down ports on which states did not change since last polling
             // We still take into account 'adsl' ports that may update speed/noise even if the interface is status down
             d_echo("$port_id skipped because port is still down since last polling.");

If this warrants a PR, I can add one.

Hello @wrf42
What did you try to achieve ? VDSL ports are not supported anyway for statistics. LibreNMS only polls ADSL mib (which incidently is (mis)used by Cisco to post VDSL stats as well.
Bye

Hi! My Zyxel modem supports the ADSL mib but has a ifType of “vdsl” - the port is shown as “down” but I still get metrics for attenuation, line speeds, etc.

I want to graph this with librenms, so that’s what the hack is for.