Smart Application stats not being picked up

evanrich · 2 February 2020 21:06

I’m having an issue with collecting smart data that I can’t seem to figure out. Scripts and everything works fine, poller doesn’t seem to output any errors, it just doesnt’ get SMART data. Here’s the output on my media server (FreeNAS) for the smart command:

root@mediastore:/mnt/sirius/homes/snmp # /mnt/sirius/homes/snmp/smart
nvme0,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,0,0,0,0,0,0,null
nvme1,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,0,0,0,0,0,0,null
ada0,0,null,null,161,null,null,0,null,null,39,null,0,0,0,100,5620,0,0,0,0,0,0,0,0,5649
da0,0,0,null,null,null,null,null,null,null,39,0,0,0,64,null,null,20,1,0,0,6,15,0,0,18771
da1,0,0,null,null,null,null,null,null,null,39,0,0,0,74,null,null,20,1,0,0,6,15,0,0,18789
da2,0,0,null,null,null,null,null,null,null,38,0,0,0,55,null,null,20,1,0,0,6,15,0,0,18788
da3,0,0,null,null,null,null,null,null,null,38,0,0,0,59,null,null,20,1,0,0,6,15,0,0,18789
da4,0,0,null,null,null,null,null,null,null,41,0,0,0,61,null,null,20,1,0,0,6,15,0,0,18782
da5,0,0,null,null,null,null,null,null,null,37,0,0,0,64,null,null,20,1,0,0,6,15,0,0,18786
da6,0,0,null,null,null,null,null,null,null,33,0,0,0,61,null,null,20,1,0,0,6,15,0,0,18786
da7,0,0,null,null,null,null,null,null,null,41,0,0,0,58,null,null,20,1,0,0,6,15,0,0,18782
da8,0,0,null,null,null,null,null,null,null,42,0,0,0,52,null,null,20,1,0,0,6,15,0,0,18782
da9,0,0,null,null,null,null,null,null,null,39,0,0,0,48,null,null,20,1,0,0,6,15,0,0,18782
da10,0,0,null,null,null,null,null,null,null,37,0,0,0,57,null,null,20,1,0,0,6,15,0,0,18761
da11,0,0,null,null,null,null,null,null,null,34,0,0,0,63,null,null,20,1,0,0,6,15,0,0,18761
da12,0,0,null,null,null,null,null,null,null,38,0,0,0,72,null,null,20,1,0,0,6,15,0,0,15534
da13,0,0,null,null,null,null,null,null,null,39,0,0,0,73,null,null,20,1,0,0,6,15,0,0,15535
da14,0,0,null,null,null,null,null,null,null,38,0,0,0,68,null,null,20,1,0,0,6,15,0,0,15513
da15,0,0,null,null,null,null,null,null,null,38,0,0,0,50,null,null,20,1,0,0,6,15,0,0,15512
da16,0,0,null,null,null,null,null,null,null,36,0,0,0,56,null,null,20,1,0,0,6,15,0,0,15489
da17,0,0,null,null,null,null,null,null,null,33,0,0,0,87,null,null,20,1,0,0,6,15,0,0,15489

Validate.php output:

====================================
Component | Version
--------- | -------
LibreNMS  | 1.59-78-gee72fa4d3
DB Schema | 2019_12_28_180000_add_overwrite_ip_to_devices (156)
PHP       | 7.3.11-1+ubuntu18.04.1+deb.sury.org+1
MySQL     | 10.4.11-MariaDB-1:10.4.11+maria~bionic-log
RRDTool   | 1.7.0
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.9.2
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[WARN]  IPv6 is disabled on your server, you will not be able to add IPv6 devices.

When I run ./poller.php -h -d, I see the following output for applications

ups-nutRRD[last 192.168.1.54/app-ups-nut-39.rrd  --daemon unix:/var/run/rrdcached/rrdcached.sock]
RRD[update 192.168.1.54/app-ups-nut-39.rrd N:100:592:13:53.8:48:120:122.0:79 --daemon unix:/var/run/rrdcached/rrdcached.sock]
SQL[UPDATE `applications` set `app_state`=?,`app_status`=?,`timestamp`=NOW() WHERE `app_id` = ? ["OK","",39] 1.61ms]
SQL[SELECT * FROM `application_metrics` WHERE app_id=? [39] 0.77ms]
: ..SQL[UPDATE `application_metrics` set `value`=?,`value_prev`=? WHERE app_id=? && metric=? [13,14,39,"time_remaining"] 0.76ms]
U.....

SMARTSQL[select `hostname`, `overwrite_ip` from `devices` where `hostname` = ? limit 1 ["192.168.1.54"] 0.8ms]
SNMP['/usr/bin/snmpbulkwalk' '-v3' '-l' 'authPriv' '-n' "" '-a' 'SHA' '-A' 'PASSWORD' '-u' 'USER' '-x' 'AES' '-X' 'PASSWORD' '-Oqv' '-M' '/opt/librenms/mibs' 'udp:HOSTNAME:161' '.1.3.6.1.4.1.8072.1.3.2.3.1.2.5.115.109.97.114.116']
RRD[last 192.168.1.54/app-smart-63-.rrd  --daemon unix:/var/run/rrdcached/rrdcached.sock]
RRD[update 192.168.1.54/app-smart-63-.rrd N:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U --daemon unix:/var/run/rrdcached/rrdcached.sock]
RRD[last 192.168.1.54/app-smart_id9-63-.rrd  --daemon unix:/var/run/rrdcached/rrdcached.sock]
RRD[update 192.168.1.54/app-smart_id9-63-.rrd N:U --daemon unix:/var/run/rrdcached/rrdcached.sock]
SQL[UPDATE `applications` set `app_state`=?,`app_status`=?,`timestamp`=NOW() WHERE `app_id` = ? ["UNKNOWN","",63] 0.99ms]
SQL[SELECT * FROM `application_metrics` WHERE app_id=? [63] 0.74ms]
: .........................
>> Runtime for poller module 'applications': 10.2152 seconds with 23584 bytes

It does UPS stats, then moves on to “SMART” buthe then just shows nothing. One thing that I noticed, was that in the RRD update line, I see " N:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U:U" which has 25 places, but if i look at the SMART output, I see 26 places is this the reason?

If I manually run the bulkwalk line, I don’t get anything back:

/usr/bin/snmpbulkwalk -v3 -l authPriv -n "" -a SHA -A  librenms -u {user} -x AES -X {password} -Oqv -M /opt/librenms/mibs udp:192.168.1.54:161 .1.3.6.1.4.1.8072.1.3.2.3.1.2.5.115.109.97.114.116
root@librenms-7ff8bc8db6-mq58r:/opt/librenms#

SourceDoctor · 2 February 2020 22:49

Hi,
did you use the newest smart script from librenms-agent Repository?

evanrich · 2 February 2020 23:13

Yup, I even tried re-grabbing it again. I have to change the shebang line to the path to perl, because for some reason freenas doesn’t like “env perl”, but other than that it’s the same script.

evanrich · 2 February 2020 23:16

Further, If I run snmpbulkwalk locally on freenas, it works:

 /usr/local/bin/snmpbulkwalk -v3 -l authPriv -n "" -a SHA -A librenms -u user -x AES -X password -Oqv -M /opt/librenms/mibs udp:192.168.1.54:161 .1.3.6.1.4.1.8072.1.3.2.3.1.2.5.115.109.97.114.116
MIB search path: /opt/librenms/mibs
Cannot find module (SNMPv2-MIB): At line 0 in (none)
Cannot find module (IF-MIB): At line 0 in (none)
Cannot find module (IP-MIB): At line 0 in (none)
Cannot find module (TCP-MIB): At line 0 in (none)
Cannot find module (UDP-MIB): At line 0 in (none)
Cannot find module (HOST-RESOURCES-MIB): At line 0 in (none)
Cannot find module (NOTIFICATION-LOG-MIB): At line 0 in (none)
Cannot find module (DISMAN-EVENT-MIB): At line 0 in (none)
Cannot find module (DISMAN-SCHEDULE-MIB): At line 0 in (none)
Cannot find module (HOST-RESOURCES-TYPES): At line 0 in (none)
Cannot find module (MTA-MIB): At line 0 in (none)
Cannot find module (NETWORK-SERVICES-MIB): At line 0 in (none)
Cannot find module (UCD-DISKIO-MIB): At line 0 in (none)
Cannot find module (UCD-SNMP-MIB): At line 0 in (none)
Cannot find module (UCD-DEMO-MIB): At line 0 in (none)
Cannot find module (SNMP-TARGET-MIB): At line 0 in (none)
Cannot find module (NET-SNMP-AGENT-MIB): At line 0 in (none)
Cannot find module (SNMP-MPD-MIB): At line 0 in (none)
Cannot find module (SNMP-USER-BASED-SM-MIB): At line 0 in (none)
Cannot find module (SNMP-FRAMEWORK-MIB): At line 0 in (none)
Cannot find module (SNMP-VIEW-BASED-ACM-MIB): At line 0 in (none)
Cannot find module (SNMP-COMMUNITY-MIB): At line 0 in (none)
Cannot find module (IP-FORWARD-MIB): At line 0 in (none)
Cannot find module (NET-SNMP-PASS-MIB): At line 0 in (none)
Cannot find module (NET-SNMP-EXTEND-MIB): At line 0 in (none)
Cannot find module (UCD-DLMOD-MIB): At line 0 in (none)
Cannot find module (SNMP-NOTIFICATION-MIB): At line 0 in (none)
Cannot find module (SNMPv2-TM): At line 0 in (none)
Cannot find module (NET-SNMP-VACM-MIB): At line 0 in (none)
"nvme0,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,0,0,0,0,0,0,null
nvme1,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,0,0,0,0,0,0,null
ada0,0,null,null,161,null,null,0,null,null,39,null,0,0,0,100,5621,0,0,0,0,0,0,0,0,5651
da0,0,0,null,null,null,null,null,null,null,40,0,0,0,64,null,null,20,1,0,0,6,15,0,0,18773
da1,0,0,null,null,null,null,null,null,null,40,0,0,0,74,null,null,20,1,0,0,6,15,0,0,18791
da2,0,0,null,null,null,null,null,null,null,39,0,0,0,55,null,null,20,1,0,0,6,15,0,0,18790
da3,0,0,null,null,null,null,null,null,null,39,0,0,0,59,null,null,20,1,0,0,6,15,0,0,18791
da4,0,0,null,null,null,null,null,null,null,42,0,0,0,61,null,null,20,1,0,0,6,15,0,0,18784
da5,0,0,null,null,null,null,null,null,null,37,0,0,0,64,null,null,20,1,0,0,6,15,0,0,18788
da6,0,0,null,null,null,null,null,null,null,34,0,0,0,61,null,null,20,1,0,0,6,15,0,0,18788
da7,0,0,null,null,null,null,null,null,null,42,0,0,0,58,null,null,20,1,0,0,6,15,0,0,18784
da8,0,0,null,null,null,null,null,null,null,43,0,0,0,52,null,null,20,1,0,0,6,15,0,0,18785
da9,0,0,null,null,null,null,null,null,null,40,0,0,0,48,null,null,20,1,0,0,6,15,0,0,18784
da10,0,0,null,null,null,null,null,null,null,38,0,0,0,57,null,null,20,1,0,0,6,15,0,0,18763
da11,0,0,null,null,null,null,null,null,null,35,0,0,0,63,null,null,20,1,0,0,6,15,0,0,18763
da12,0,0,null,null,null,null,null,null,null,39,0,0,0,72,null,null,20,1,0,0,6,15,0,0,15536
da13,0,0,null,null,null,null,null,null,null,40,0,0,0,73,null,null,20,1,0,0,6,15,0,0,15537
da14,0,0,null,null,null,null,null,null,null,39,0,0,0,68,null,null,20,1,0,0,6,15,0,0,15515
da15,0,0,null,null,null,null,null,null,null,39,0,0,0,50,null,null,20,1,0,0,6,15,0,0,15515
da16,0,0,null,null,null,null,null,null,null,37,0,0,0,56,null,null,20,1,0,0,6,15,0,0,15492
da17,0,0,null,null,null,null,null,null,null,33,0,0,0,87,null,null,20,1,0,0,6,15,0,0,15492"

root@mediastore:/mnt/sirius/homes/snmp #

I’ve tried disabling/re-enabling the SMART app in librenms as well, didn’t work. The strange thing is, it works fine on other servers, just the freenas box has an issue.

SourceDoctor · 5 February 2020 20:24

i tried it with your output on a testsystem,
graphing works well.
Not sure what’s the reason for your issue

evanrich · 18 March 2020 05:30

So I’ve made a little progress, still not sure what the root cause is, but I’ve enabled tcp mode on freenas,and now it seems to be collecting metrics, at least when I run “poller”

SMART
SNMP['/usr/bin/snmpbulkwalk' '-v2c' '-c' 'COMMUNITY' '-Oqv' '-M' '/opt/librenms/mibs' 'tcp:HOSTNAME:161' '.1.3.6.1.4.1.8072.1.3.2.3.1.2.5.115.109.97.114.116']

"da0,0,0,null,null,null,null,null,null,null,39,0,0,0,64,null,null,20,1,0,0,6,15,0,0,19835
da1,0,0,null,null,null,null,null,null,null,39,0,0,0,74,null,null,20,1,0,0,6,15,0,0,19854
da2,0,0,null,null,null,null,null,null,null,39,0,0,0,55,null,null,20,1,0,0,6,15,0,0,19852
da3,0,0,null,null,null,null,null,null,null,38,0,0,0,59,null,null,20,1,0,0,6,15,0,0,19853
da4,0,0,null,null,null,null,null,null,null,41,0,0,0,61,null,null,20,1,0,0,6,15,0,0,19847
da5,0,0,null,null,null,null,null,null,null,36,0,0,0,64,null,null,20,1,0,0,6,15,0,0,19850
da6,0,0,null,null,null,null,null,null,null,34,0,0,0,61,null,null,20,1,0,0,6,15,0,0,19850
da7,0,0,null,null,null,null,null,null,null,40,0,0,0,58,null,null,20,1,0,0,6,15,0,0,19847
da8,0,0,null,null,null,null,null,null,null,41,0,0,0,52,null,null,20,1,0,0,6,15,0,0,19847
da9,0,0,null,null,null,null,null,null,null,38,0,0,0,48,null,null,20,1,0,0,6,15,0,0,19847
da10,0,0,null,null,null,null,null,null,null,37,0,0,0,57,null,null,20,1,0,0,6,15,0,0,19825
da11,0,0,null,null,null,null,null,null,null,33,0,0,0,63,null,null,20,1,0,0,6,15,0,0,19825
da12,0,0,null,null,null,null,null,null,null,38,0,0,0,72,null,null,20,1,0,0,6,15,0,0,16598
da13,0,0,null,null,null,null,null,null,null,39,0,0,0,73,null,null,20,1,0,0,6,15,0,0,16599
da14,0,0,null,null,null,null,null,null,null,38,0,0,0,68,null,null,20,1,0,0,6,15,0,0,16577
da15,0,0,null,null,null,null,null,null,null,37,0,0,0,50,null,null,20,1,0,0,6,15,0,0,16577
da16,0,0,null,null,null,null,null,null,null,35,0,0,0,56,null,null,20,1,0,0,6,15,0,0,16554
da17,0,0,null,null,null,null,null,null,null,32,0,0,0,87,null,null,20,1,0,0,6,15,0,0,16554"

however stats still aren’t showing up…yet. Will continue to monitor

tfl · 26 October 2020 22:15

Hi!
It seems to i have same problem…
But in any case i have workaround:
I’m added to URL this /disk=da13/ and voila!

https://LIBRENMSSITE.com/device/device=YOUR_CURRENT_DEVICE/tab=apps/app=smart/disk=YOUR_DEVICE_NAME_FROM_SMART/

Primer in my case

http://10.10.10.10/device/device=9/tab=apps/app=smart/disk=sda/

system · 12 October 2021 13:15