Mounting ISO image in Windows server causing incorrect disc space data enumeration

Ok here’s a very weird bug.

A number of the servers we monitor in LibreNMS are Windows servers, various versions from 2012 to 2019. We just use the built in SNMP server in Windows and it has been working great for over a year.

I received a low disk space alert to tell me that a data drive mounted as I:\ which is normally 2TB in size with 1.5TB free was 100% full, and that its total size was 1.4GB… :thinking:

Checking with a college I discovered that he had mounted an iso DVD image, which mounted as E:\

LibreNMS had started mapping the disk size and free space values from the DVD in E drive onto the data volume on I drive…

I tried running both Discovery and Polling manually while this DVD was inserted however it made no difference. Here it is in the disk space summary:

And here is part of the alert I received which lists the exact size it believes the volume had suddenly become:

Drive: I:\ Label:PASS DATA  Serial Number 48f0ceb6

Disk Utilization: 100%
Disk Size: 1.44 GB
Disk Free: 0.00 GB

Here is the SNMP data while the virtual DVD was mounted:

hrStorageIndex.1 = 1
hrStorageIndex.2 = 2
hrStorageIndex.3 = 3
hrStorageIndex.4 = 4
hrStorageIndex.5 = 5
hrStorageIndex.6 = 6
hrStorageType.1 = hrStorageFixedDisk
hrStorageType.2 = hrStorageCompactDisc
hrStorageType.3 = hrStorageCompactDisc
hrStorageType.4 = hrStorageFixedDisk
hrStorageType.5 = hrStorageVirtualMemory
hrStorageType.6 = hrStorageRam
hrStorageDescr.1 = C:\ Label:  Serial Number 18b508b8
hrStorageDescr.2 = D:\
hrStorageDescr.3 = E:\ Label:SqlSetup_x64_ENU  Serial Number f1775a81
hrStorageDescr.4 = I:\ Label:PASS DATA  Serial Number 48f0ceb6
hrStorageDescr.5 = Virtual Memory
hrStorageDescr.6 = Physical Memory
hrStorageAllocationUnits.1 = 4096
hrStorageAllocationUnits.2 = 0
hrStorageAllocationUnits.3 = 2048
hrStorageAllocationUnits.4 = 4096
hrStorageAllocationUnits.5 = 65536
hrStorageAllocationUnits.6 = 65536
hrStorageSize.1 = 134072575
hrStorageSize.2 = 0
hrStorageSize.3 = 755002
hrStorageSize.4 = 524287231
hrStorageSize.5 = 1204207
hrStorageSize.6 = 1048559
hrStorageUsed.1 = 12730002
hrStorageUsed.2 = 0
hrStorageUsed.3 = 755002
hrStorageUsed.4 = 111504966
hrStorageUsed.5 = 376651
hrStorageUsed.6 = 363907
hrStorageAllocationFailures.1 = 0
hrStorageAllocationFailures.2 = 0
hrStorageAllocationFailures.3 = 0
hrStorageAllocationFailures.4 = 0
hrStorageAllocationFailures.5 = 0
hrStorageAllocationFailures.6 = 0  

After the virtual DVD was ejected everything seems back to normal again, the alert is cleared and the disc space graphs are correct, and here is part of the recovery alert confirming this is showing the correct sizes again:

Drive: I:\ Label:PASS DATA  Serial Number 48f0ceb6

Disk Utilization: 21%
Disk Size: 2,000.00 GB
Disk Free: 1,574.64 GB

Here is another SNMP capture after the DVD is removed:

hrStorageIndex.1 = 1
hrStorageIndex.2 = 2
hrStorageIndex.3 = 3
hrStorageIndex.4 = 4
hrStorageIndex.5 = 5
hrStorageType.1 = hrStorageFixedDisk
hrStorageType.2 = hrStorageCompactDisc
hrStorageType.3 = hrStorageFixedDisk
hrStorageType.4 = hrStorageVirtualMemory
hrStorageType.5 = hrStorageRam
hrStorageDescr.1 = C:\ Label:  Serial Number 18b508b8
hrStorageDescr.2 = D:\
hrStorageDescr.3 = I:\ Label:PASS DATA  Serial Number 48f0ceb6
hrStorageDescr.4 = Virtual Memory
hrStorageDescr.5 = Physical Memory
hrStorageAllocationUnits.1 = 4096
hrStorageAllocationUnits.2 = 0
hrStorageAllocationUnits.3 = 4096
hrStorageAllocationUnits.4 = 65536
hrStorageAllocationUnits.5 = 65536
hrStorageSize.1 = 134072575
hrStorageSize.2 = 0
hrStorageSize.3 = 524287231
hrStorageSize.4 = 1204207
hrStorageSize.5 = 1048559
hrStorageUsed.1 = 12723863
hrStorageUsed.2 = 0
hrStorageUsed.3 = 111505110
hrStorageUsed.4 = 375187
hrStorageUsed.5 = 362758
hrStorageAllocationFailures.1 = 0
hrStorageAllocationFailures.2 = 0
hrStorageAllocationFailures.3 = 0
hrStorageAllocationFailures.4 = 0
hrStorageAllocationFailures.5 = 0

Comparing the two SNMP captures it’s apparent that when you mount an ISO image in windows server to a drive letter using the in built iso mounter that it inserts itself into the hrStorage list in the same order as the drive letters, so mounting a transient drive like E (which didn’t exist at all prior to mounting the iso) into a gap before fixed drives bumps the device on I drive from entity 3 to entity 4 in the SNMP list.

However while the new data provided by SNMP is internally consistent, (names, sizes etc all move to entity 4 and match up with each other) it seems LibreNMS can’t cope with this, with the description of the drive remaining the same, but the size data coming from the same entity number which is now the virtual DVD.

I don’t understand why running both Discovery and Poller didn’t straighten this out either.

Is it possible for this to be fixed somehow so that it can deal with dynamic insertions/deletions of drive indexes on a server ?

Steps to reproduce this should be fairly easy:

  1. Mount a fixed disk volume to a “high” drive letter leaving a gap in drive letters, poll with LibreNMS to establish storage graphs, optionally set up a low disc space alert.

  2. Mount a DVD ISO image on the server which will mount to a drive letter before the fixed volume, creating a drive letter that didn’t exist before. (Not just an empty device like a real DVD drive with no disc inserted) This should cause the drive indexes in SNMP to be bumped along and incorrect size data will now be shown in LibreNMS.

[email protected]:~$ ./validate.php
====================================
Component | Version
--------- | -------
LibreNMS  | 21.11.0
DB Schema | 2021_11_12_123037_change_cpwVcID_to_unsignedInteger (225)
PHP       | 7.4.3
Python    | 3.8.10
MySQL     | 10.3.32-MariaDB-0ubuntu0.20.04.1
RRDTool   | 1.7.2
SNMP      | NET-SNMP 5.8
====================================

[OK]    Composer Version: 2.1.14
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[WARN]  Your database schema has extra migrations (2021_11_12_123037_change_cpwVcID_to_unsignedInteger). If you just switched to the stable release from the daily release, your database is in between releases and this will be resolved with the next release.
[FAIL]  Database: incorrect column (pseudowires/cpwVcID)
[FAIL]  We have detected that your database schema may be wrong
        [FIX]:
        Run the following SQL statements to fix it
        SQL Statements:
         ALTER TABLE `pseudowires` CHANGE `cpwVcID` `cpwVcID` int NOT NULL ;

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.