Issues with Resolved IP address

1

Hi All,

one fine day i saw that some devices changed their IPs to 127.0.0.53 in NMS.
the log record is: 2016-10-12 07:15:32 System Device IP changed to 127.0.53.53
after this had happened NMS can’t get information from theses devices,
i tried to rediscover device without success - the device IPs are 127.0.53.53
local ping to the devices shows hostname exists in the DNS (/etc/hosts).
i tried to rename device (./renamehost.php) - without success.
i see i can delete/add devices from NMS to resolve this issue, but there are hundreds devices with specific setting and it will take long to do it.

how can i say to NMS to rediscover device’s IPs ?

2

Your post is a bit confusing.

The change in IP is because we detected that the hostname resolves to a different IP from last time we checked (every 5 minutes in polling). However the IP changing is no issue for us, we don’t use it just merely log it.

We can only assume you add devices by hostname as you’ve not specifically said that. However the devices themselves don’t rename themselves so if you add them by IP they will always be contacted using that IP.

I suggest you run ./validate.php then ./poller.php -h HOSTNAME -r -f -d -m os and pastebin the output

3

I see,

according the output below the NMS cannot get some OIDs due to missing of the MIB file ?

[root@nms librenms]# ./validate.php

LibreNMS Version: 46616c13927577d2408a2df31fadb325f6e92045
DB Schema: 148
PHP: 7.0.11
MySQL: 5.5.50-MariaDB
RRDTool: 1.4.8
SNMP: NET-SNMP 5.7.2

[WARN] You don’t have $config[“user”] set, this most likely needs to be set to librenms
[OK] Database connection successful
[FAIL] You have no timezone set for php: PHP: Runtime Configuration - Manual
[root@nms librenms]# ./poller.php -h 3000a.bdy01.pas01.radio -r -f -d -m os
Southland Communications Poller
Version info:
Commit SHA: 46616c13927577d2408a2df31fadb325f6e92045
Commit Date: 1476993241
DB Schema: 148
PHP: 7.0.11
MySQL: 5.5.50-MariaDB
RRDTool: 1.4.8
SNMP: NET-SNMP 5.7.2
DEBUG!
Starting polling run:

SQL[SELECT device_id FROM devices WHERE disabled = 0 AND hostname = ‘3000a.bdy01.pas01.radio’ ORDER BY device_id ASC]
Hostname: 3000a.bdy01.pas01.radio
Device ID: 33
OS: dragonwavec

SNMP[/usr/bin/snmpget -v2c -c COMMUNITY -Oqv -m SNMPv2-MIB -M /opt/librenms/mibs udp:HOSTNAME:161 sysObjectID.0]
SNMPv2-SMI::enterprises.7262.2.2

SNMP[/usr/bin/snmpget -v2c -c COMMUNITY -OQnUst -m SNMPv2-MIB:HOST-RESOURCES-MIB:SNMP-FRAMEWORK-MIB -M /opt/librenms/mibs udp:HOSTNAME:161 sysUpTime.0 sysLocation.0 sysContact.0 sysName.0 sysObjectID.0]
sysUpTime.0 = 2440531201
sysLocation.0 = Bradley PCN/RSL -36.5 Typ -36.2
sysContact.0 =
sysName.0 = Pasadena to Bradley #3 H Dish #2
sysObjectID.0 = enterprises.7262.2.2

SNMP[/usr/bin/snmpget -v2c -c COMMUNITY -OvQ -m SNMPv2-MIB:HOST-RESOURCES-MIB:SNMP-FRAMEWORK-MIB -M /opt/librenms/mibs udp:HOSTNAME:161 sysDescr.0]
hc50_322_256qam Omni: 1.04.08

SNMP[/usr/bin/snmpget -v2c -c COMMUNITY -OQnUst -m HOST-RESOURCES-MIB:SNMP-FRAMEWORK-MIB -M /opt/librenms/mibs udp:HOSTNAME:161 snmpEngineTime.0 hrSystemUptime.0]
snmpEngineTime.0 = 16713378
hrSystemUptime.0 = No Such Object available on this agent at this OID

Using SNMP Agent Uptime (24405312s)
RRD[update /opt/librenms/rrd/3000a.bdy01.pas01.radio/uptime.rrd N:24405312]
[RRD Disabled]Uptime: 282 days, 11h 15m 12s
Using cached lat/lng from other device

Load poller module os

SNMP[/usr/bin/snmpget -v2c -c COMMUNITY -OQv udp:HOSTNAME:161 HORIZON-ODU-MIB::hzOduUnitSerialNo.0]
MIB search path: /root/.snmp/mibs:/usr/share/snmp/mibs
Cannot find module (HORIZON-ODU-MIB): At line 0 in (none)
HORIZON-ODU-MIB::hzOduUnitSerialNo.0: Unknown Object Identifier

SNMP[/usr/bin/snmpget -v2c -c COMMUNITY -OQv udp:HOSTNAME:161 SNMPv2-MIB::sysLocation.0]
Bradley PCN/RSL -36.5 Typ -36.2

SNMP[/usr/bin/snmpget -v2c -c COMMUNITY -OQv udp:HOSTNAME:161 HORIZON-ODU-MIB::hzOduSystemLicensedSpeed.0]
MIB search path: /root/.snmp/mibs:/usr/share/snmp/mibs
Cannot find module (HORIZON-ODU-MIB): At line 0 in (none)
HORIZON-ODU-MIB::hzOduSystemLicensedSpeed.0: Unknown Object Identifier

SNMP[/usr/bin/snmpget -v2c -c COMMUNITY -OQv udp:HOSTNAME:161 SNMPv2-MIB::sysDescr.0]
hc50_322_256qam Omni: 1.04.08

SNMP[/usr/bin/snmpget -v2c -c COMMUNITY -OQv udp:HOSTNAME:161 HORIZON-ODU-MIB::hzOduModem1ChannelizedRslBelowThresholdParameters.0]
MIB search path: /root/.snmp/mibs:/usr/share/snmp/mibs
Cannot find module (HORIZON-ODU-MIB): At line 0 in (none)
HORIZON-ODU-MIB::hzOduModem1ChannelizedRslBelowThresholdParameters.0: Unknown Object Identifier

SNMP[/usr/bin/snmpget -v2c -c COMMUNITY -OQv udp:HOSTNAME:161 HORIZON-ODU-MIB::hzOduIpAddress.0]
MIB search path: /root/.snmp/mibs:/usr/share/snmp/mibs
Cannot find module (HORIZON-ODU-MIB): At line 0 in (none)
HORIZON-ODU-MIB::hzOduIpAddress.0: Unknown Object Identifier

Hardware:
Version: 1.04.08
Features: WEB: Bradley PCN/RSL -36.5 Typ -36.2 SPEED: Mbit MODE: hc50_322_256qam THRESHOLD: dBm
Serial:

Runtime for poller module ‘os’: 0.11609601974487 seconds

Unload poller module os

RRD[update /opt/librenms/rrd/3000a.bdy01.pas01.radio/poller-perf-os.rrd N:0.11609601974487]
[RRD Disabled]### Start Device Groups ###
SQL[SELECT DISTINCT(devices.device_id) FROM devices WHERE devices.device_id=‘33’ AND (devices.status = “down” ) LIMIT 1]
SQL[SELECT DISTINCT(alerts.device_id) FROM alerts WHERE alerts.device_id=‘33’ AND (alerts.rule_id = “10” && alerts.state = “1” ) LIMIT 1]
SQL[SELECT DISTINCT(alerts.device_id) FROM alerts WHERE alerts.device_id=‘33’ AND (alerts.rule_id = “7” && alerts.state = “1” ) LIMIT 1]
SQL[SELECT device_group_id FROM device_group_device WHERE device_id=‘33’]
Groups Added:
Groups Removed:

End Device Groups

RRD[update /opt/librenms/rrd/3000a.bdy01.pas01.radio/poller-perf.rrd N:3.293]
[RRD Disabled]Polled in 3.293 seconds
Updating 3000a.bdy01.pas01.radio
SQL[UPDATE devices set uptime =‘24405312’,last_polled =NOW(),last_polled_timetaken =‘3.293’ WHERE device_id = ‘33’]
UPDATED!

Start Alerts

SQL[SELECT device_group_id FROM device_group_device WHERE device_id=‘33’]
SQL[SELECT alert_schedule.schedule_id FROM alert_schedule LEFT JOIN alert_schedule_items ON alert_schedule.schedule_id=alert_schedule_items.schedule_id WHERE ( alert_schedule_items.target = ‘33’ ) && NOW() BETWEEN alert_schedule.start AND alert_schedule.end LIMIT 1]
SQL[SELECT device_group_id FROM device_group_device WHERE device_id=‘33’]
Rule #1 (Device Unreachable):
SQL[SELECT * FROM devices WHERE (devices.device_id = ‘33’) && ( ( ( devices.status = 0 && ( ( devices.disabled = 0 && devices.ignore = 0 ) ) ) ) = “1” )]
Status: NOCHG
Rule #2 (Device rebooted):
SQL[SELECT * FROM devices WHERE (devices.device_id = ‘33’) && (devices.uptime < “300” && ( ( devices.disabled = 0 && devices.ignore = 0 ) ) = “1” )]
Status: NOCHG
Rule #3 (BGP Session down):
SQL[SELECT * FROM bgpPeers,devices WHERE (( devices.device_id = bgpPeers.device_id ) && bgpPeers.device_id = ‘33’) && (bgpPeers.bgpPeerState != “established” && ( ( devices.status = 1 && ( ( devices.disabled = 0 && devices.ignore = 0 ) ) ) ) = “1” )]
Status: NOCHG
Rule #4 (BGP Session establised):
SQL[SELECT * FROM bgpPeers WHERE (bgpPeers.device_id = ‘33’) && (bgpPeers.bgpPeerFsmEstablishedTime < “300” && bgpPeers.bgpPeerState = “established” )]
Status: NOCHG
Rule #7 (Wireless Link Down):
SQL[SELECT * FROM sensors,devices WHERE (( devices.device_id = sensors.device_id ) && sensors.device_id = ‘33’) && (sensors.sensor_current > sensors.sensor_limit && devices.type = “Wireless” && sensors.sensor_index = “WirelessLink” )]
Status: NOCHG

End Alerts

SQL[INSERT INTO perf_times (type,doing,start,duration,devices,poller) VALUES (‘poll’,‘3000a.bdy01.pas01.radio’,‘1477031137.7362’,‘3.316’,‘1’,‘nms.southlandcommunications.com\n’)]
./poller.php 3000a.bdy01.pas01.radio 2016-10-21 06:25:41 - 1 devices polled in 3.316 secs

MySQL: Cell[4/0s] Row[4/0s] Rows[8/0s] Column[3/0s] Update[1/0s] Insert[1/0s] Delete[0/0s]

4

You have a warn and a fail you need to fix in validate. I suggest you do that.

The poller completed ok so that device isn’t down, try it on one that is.

Please use pastebin next time, a wall of text is annoying to try and read.

5

Thank you,

i see there isn’t ip problem