Random segmentation faults on mempool discover module

I’m facing problems with Segmentation faults on the mempool functionality. I already did a bugreport on github: https://github.com/librenms/librenms/issues/13214 . But seems that’s not the correct way :thinking:

The problem

I moved my LibreNMS instance to a new server (RHEL 7 to RHEL 8) everything runs ok. But the discovery (PHP actually) process is constantly segfaulting on the mempool module on random hosts.

Also the memory graphs seems not be filled.

When I disable the Mempool module it does not segfaults.

So far I tried:

  • Disabling SElinux
  • Switched between PHP 7.3, 7.4 and 8.0
  • Disable/Enable Mempool

Output of ./validate.php

====================================
Component | Version
--------- | -------
LibreNMS  | 21.8.0-41-g0a76ca444
DB Schema | 2021_25_01_0129_isis_adjacencies_nullable (217)
PHP       | 7.3.29
Python    | 3.6.8
MySQL     | 10.6.4-MariaDB
RRDTool   | 1.7.0
SNMP      | NET-SNMP 5.8
====================================

[OK]    Composer Version: 2.1.6
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct```

What was the last working version of LibreNMS?

No response

Anything in the logs that might be useful for us?

While running with -d and Xdebug (xdebug only enabled because of the segfault!) these are the last lines:

SNMP['/usr/bin/snmpget' '-v2c' '-c' 'COMMUNITY' '-OQUv' '-m' 'HOST-RESOURCES-MIB' '-M' '/var/path/website/html/webroot/librenms/mibs' 'udp:HOSTNAME:161' 'hrMemorySize.0']
3823388  
  
Free memory adjusted by availability calculation: 1004.2 MiiB -> 2 GiiB
  
SQL[select * from `mempools` where `mempools`.`device_id` = ? and `mempools`.`device_id` is not null [628] 0.53ms] 
  
SQL[update `mempools` set `mempool_perc` = ?, `mempool_used` = ?, `mempool_used_oid` = ?, `mempool_free` = ? where `mempool_id` = ? [45,1764937728,".1.3.6.1.2.1.25.2.3.1.6.1",2150211584,1753] 1.41ms] 
  
Updated data:  
array (
  'mempool_perc' => 45.0,
  'mempool_used' => 1764937728,
  'mempool_used_oid' => '.1.3.6.1.2.1.25.2.3.1.6.1',
  'mempool_free' => 2150211584,
)  
SQL[update `mempools` set `mempool_class` = ?, `mempool_perc` = ?, `mempool_used` = ?, `mempool_used_oid` = ?, `mempool_free` = ? where `mempool_id` = ? ["virtual",46,3739041792,".1.3.6.1.2.1.25.2.3.1.6.3",4471070720,1754] 0.97ms] 
  
Updated data:  
array (
  'mempool_class' => 'virtual',
  'mempool_perc' => 46.0,
  'mempool_used' => 3739041792,
  'mempool_used_oid' => '.1.3.6.1.2.1.25.2.3.1.6.3',
  'mempool_free' => 4471070720,
)  
Mempool class changed Virtual memory (1754)  
Xdebug has detected a possible infinite loop, and aborted your script with a stack depth of '256' frames {"exception":"[object] (Error(code: 0): Xdebug has detected a possible infinite loop, and aborted your script with a stack depth of '256' frames at /var/path/website/html/webroot/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php:1322)"}

Why do you have xdebug enabled?

Because it gave more details about the segfault and I enabled it on purpose for this debug case. Needless to say I did not had it enabled initially when the segfaults arise.

Can you give the entire mempool section output. You cut off the top which has relevant information (the snmp data)

Sure @murrant , thanks for aksing!

communityStringIndexing  
SNMP['/usr/bin/snmpbulkwalk' '-v2c' '-c' 'COMMUNITY' '-OQUsetX' '-m' 'HOST-RESOURCES-MIB:HOST-RESOURCES-TYPES' '-M' '/var/path/librenms/mibs' 'udp:HOSTNAME:161' 'hrStorageTable']
hrStorageIndex[1] = 1
hrStorageIndex[3] = 3
hrStorageIndex[6] = 6
hrStorageIndex[7] = 7
hrStorageIndex[8] = 8
hrStorageIndex[10] = 10
hrStorageIndex[35] = 35
hrStorageIndex[37] = 37
hrStorageIndex[38] = 38
hrStorageIndex[52] = 52
hrStorageIndex[58] = 58
hrStorageIndex[59] = 59
hrStorageType[1] = hrStorageRam
hrStorageType[3] = hrStorageVirtualMemory
hrStorageType[6] = hrStorageOther
hrStorageType[7] = hrStorageOther
hrStorageType[8] = hrStorageOther
hrStorageType[10] = hrStorageVirtualMemory
hrStorageType[35] = hrStorageFixedDisk
hrStorageType[37] = hrStorageFixedDisk
hrStorageType[38] = hrStorageFixedDisk
hrStorageType[52] = hrStorageFixedDisk
hrStorageType[58] = hrStorageFixedDisk
hrStorageType[59] = hrStorageFixedDisk
hrStorageDescr[1] = Physical memory
hrStorageDescr[3] = Virtual memory
hrStorageDescr[6] = Memory buffers
hrStorageDescr[7] = Cached memory
hrStorageDescr[8] = Shared memory
hrStorageDescr[10] = Swap space
hrStorageDescr[35] = /dev/shm
hrStorageDescr[37] = /run
hrStorageDescr[38] = /sys/fs/cgroup
hrStorageDescr[52] = /
hrStorageDescr[58] = /boot
hrStorageDescr[59] = /var
hrStorageAllocationUnits[1] = 1024
hrStorageAllocationUnits[3] = 1024
hrStorageAllocationUnits[6] = 1024
hrStorageAllocationUnits[7] = 1024
hrStorageAllocationUnits[8] = 1024
hrStorageAllocationUnits[10] = 1024
hrStorageAllocationUnits[35] = 4096
hrStorageAllocationUnits[37] = 4096
hrStorageAllocationUnits[38] = 4096
hrStorageAllocationUnits[52] = 4096
hrStorageAllocationUnits[58] = 4096
hrStorageAllocationUnits[59] = 4096
hrStorageSize[1] = 1881992
hrStorageSize[3] = 6076292
hrStorageSize[6] = 1881992
hrStorageSize[7] = 1061444
hrStorageSize[8] = 140520
hrStorageSize[10] = 4194300
hrStorageSize[35] = 235249
hrStorageSize[37] = 235249
hrStorageSize[38] = 235249
hrStorageSize[52] = 2031470
hrStorageSize[58] = 124914
hrStorageSize[59] = 9642848
hrStorageUsed[1] = 1670772
hrStorageUsed[3] = 1685176
hrStorageUsed[6] = 136796
hrStorageUsed[7] = 1061444
hrStorageUsed[8] = 140520
hrStorageUsed[10] = 14404
hrStorageUsed[35] = 1
hrStorageUsed[37] = 4229
hrStorageUsed[38] = 0
hrStorageUsed[52] = 756847
hrStorageUsed[58] = 47347
hrStorageUsed[59] = 2923288  
  
SNMP['/usr/bin/snmpget' '-v2c' '-c' 'COMMUNITY' '-OQUv' '-m' 'HOST-RESOURCES-MIB' '-M' '/var/path/librenms/mibs' 'udp:HOSTNAME:161' 'hrMemorySize.0']
1881992  
  
Free memory adjusted by availability calculation: 206.27 MiiB -> 1.34 GiiB
  
SQL[select * from `mempools` where `mempools`.`device_id` = ? and `mempools`.`device_id` is not null [109] 0.56ms] 
  
SQL[update `mempools` set `mempool_perc` = ?, `mempool_used` = ?, `mempool_used_oid` = ?, `mempool_free` = ? where `mempool_id` = ? [25,483872768,".1.3.6.1.2.1.25.2.3.1.6.1",1443287040,337] 1.45ms] 
  
Updated data:  
array (
  'mempool_perc' => 25.0,
  'mempool_used' => 483872768,
  'mempool_used_oid' => '.1.3.6.1.2.1.25.2.3.1.6.1',
  'mempool_free' => 1443287040,
)  
SQL[update `mempools` set `mempool_class` = ?, `mempool_perc` = ?, `mempool_used` = ?, `mempool_used_oid` = ?, `mempool_free` = ? where `mempool_id` = ? ["virtual",28,1725620224,".1.3.6.1.2.1.25.2.3.1.6.3",4496502784,338] 0.81ms] 
  
Updated data:  
array (
  'mempool_class' => 'virtual',
  'mempool_perc' => 28.0,
  'mempool_used' => 1725620224,
  'mempool_used_oid' => '.1.3.6.1.2.1.25.2.3.1.6.3',
  'mempool_free' => 4496502784,
)  
Mempool class changed Virtual memory (338)

Screenshot of LibreNMS memory usage (used and used perc are both zero) :

@murrant I somehow “fixed” this, by truncating the mempools table (not the structure!). After the truncate and a new discover.php run, it seems to run fine and the graphs are also being filled again.

So I guess things got corrupted somehow …

Odd how you got in a bad state… maybe there was two with the same key somehow or something.

Glad you got it resolved.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.