FIXED: Device OS changed ios => ibmtl

Help
1

Hello *,

I urgently need help with my system here.

This night, starting 2017-10-27 00:18:31

$ tail daily.log
Device performance times cleared for entries over 7 days
Returned: 0
Fetching notifications
[ Fri, 27 Oct 2017 00:15:58 +0200 ] LibreNMS Notifications (1)
[ Fri, 27 Oct 2017 00:15:58 +0200 ] misc/notifications.rss (8)
[ Fri, 27 Oct 2017 00:15:58 +0200 ] Updating DB Done
Returned: 0
Caching PeeringDB data

Returned: 0

all of my IOS devices have change to type storage and and OS=ibmtl
and
my “localhost” has changed from “server” to “network”.

A handful of my 182 IOS-devices have changed back to “network” again.
For my C6509 this happened after I manually started a “rediscover”, for other devices this happened by itself.

On most of my devices, even when starting a “rediscover” they did not change.

Could it be possible that some code change is causing this problem?

How can I help to debug the issue?

$ ./validate.php

Component Version
LibreNMS 1.32-99-gd4b2f6e
DB Schema 212
PHP 7.0.23
MySQL 5.5.56-MariaDB
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2
====================================

[OK] Database connection successful
[OK] Database schema correct
[FAIL] The poller (vls-10-nms
) has not complete within the last 5 minutes, check the cron job
[FAIL] Discovery has not completed in the last 24 hours.
[FIX] Check the cron job to make sure it is running and using discovery-wrapper.py

Thank you!

Best regards,
awaum

2 Likes
2

im seeing the same thing. @murrant

3

Please run:

./discovery.php -h all -m os

and see if that corrects the issue.

1 Like
4

@murrant Thank you for your reply!

Well, actually it solved all issues for the moment.
All IOS devices have become device type “network” again, the OS has become “ios” again and I can also see all OS details again (was checking for a certain version… this is how I discovered the issue, as the version was not shown).

“My libre” has become device type “server” again.

I had one persisting issue which turned out as a wrong snmp configuration on the switch… no idea why I thought it was working before…
However, after fixing this and another ./discovery.php on that particular device solved the issue.

Thank you for your help!
… but have you got any idea what has caused the change of OS …?

Best regards,
awaum

5

I also am recovering from this - many devices changed to ibmtl for no reason

6

I am in the same boat. I have tried to perform a re-discovery of the OS for all of my nodes (as suggested by murrant), it worked but it appears to be coming back now to ’ ibmtl '.

7

Likewise Here - Just Bumping the thread.

8

run ./daliy.sh

and @Syn2

9

After running ./daily.sh I did the following

sudo rm /opt/librenms/cache/os_defs.cache
then right after that I ran

./discovery.php -h all -m os -d

fixed it for me. :slight_smile:

2 Likes
10

Is anyone still experiencing this and if so, what version are you currently on?

11

All well Here laf. Seems the update on Friday fixed it.

Many thanks to the LibreNMS Team and Community.

1 Like
12

Back to work after a long long weekend (thank you, Martin Luther!)…

Yes, everything still looks good, my IOS devices are no longer tape libraries.

Thanks to the team for the quick response and fix!

2 Likes
13

Unfortunately the problem appeared again.
What is even more strange, for about only half of my devices (IOS and NX-OS)…

First I ran a daily.sh… but that did not change anything.
After that, I ran

./discovery.php -h all -m os -d

which changed byack OS to the appropreate, but still left the device_type on “storage”.

I followed @Kevin_Krumm 's direction and ran

sudo rm /opt/librenms/cache/os_defs.cache

followed by another

./discovery.php -h all -m os -d

While the discovery.php was still running, I could already see that all my CISCO devices changed back to device_type “network” again…

Now everything looks good again, but I’m afraid that there still might be a flaw in the code…

Any idea?

Thank you!
-awaum

14

====================================

Component Version
LibreNMS 1.33-31-gaef1a5f
DB Schema 215
PHP 7.0.24
MySQL 5.5.56-MariaDB
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2
====================================
15

Yes even i am facing same issue now and most of the devices have turned to windows or tape library in os section

17

I really need a debug of the discovery output of a device that incorrectly detects as ibmtl.

./discovery.php -h HOSTNAME -m os -d

18

@murrant I really would like to provide this debug output.
The problem is, that right now everything is OK again, after I have run

./discovery.php -h HOSTNAME -m os -d

… in other words:
When I run this manually, it correctly discovers the OS.

The problem arose during last night… there some process (right after 00:18 (daily.sh…) and the next 5 hours) changed half of my IOS or NX-OS to “ibmtl” …

Is there some difference between “what happens at night” and when I run the above command in order to solve the issue again…?

Best regards,
-awaum

19

Yeah, we have been having trouble tracking down this issue. I think it might have to do with a stale cache still even though we tried to fix that already.

Thanks for trying.

20

Murrant - would it help if I could get you access to my librenms?

21

Now it thinks the device is “generic”. This is not good. Here is the output of the discovery.

https://pastebin.com/YfHU0VLM