Bringing unix-agent <<< mem >>> stats into unix os devices

I’m sharing some of the frustrations of the community on the shortcomings of how linux memory is delivered in SNMP:

Sources

Linux Memory Pools reporting wrong

Making Physical Memory graphs valid and worthwile

Memory Pool

Linux - Memory usage confusion on Front page. · Issue #3179 · librenms/librenms · GitHub

Memory Health/Occupation Linux Machines: cached memory is not considered free/available · Issue #5660 · librenms/librenms · GitHub

Ideally, Net-SNMP would deliver the proper “Used” memory that we are familiar with from free -h, BUT having to rewrite code in a project like that seems like a lot of work :slight_smile: . Instead, for anyone running the unix-agent (check_mk), memory stats are already being collected and sent to your LibreNMS server, but nothing is being done with it. I would like to start working on a way to include those stats.

I would like some guidance on how to include it. Looking at ./includes/polling/unix-agent.inc.php, there are scripts included from ./includes/polling/unix-agent/ and from ./includes/polling/applications/. Should I write an application module for “mempool”, that would show up under the Apps tab of a device, or write a unix-agent submodule to be used in another module (probably os/unix.inc.php, similar to ‘dmi’)? I think it would be best to not write it to be included in the SNMP mempools section of the device pages, as to signify that the info is being collected differently (SNMP vs check_mk).

Thanks, and let me know any suggestions

I’ve just started using librenms and very impressed I am. But, I like the gentleman above like my memory reporting to be right. I can accept right is a little subjective on this one. I will expand. I do not like my memory to be red on all my boxes simply because Linux is making best use of memory that would otherwise not be doing anything. As a sysadmin, this was a deal breaker for me.

So I fixed it, at the moment badly, but I might do a better job, depending on what others have to say. I have limited grasp of how this thing hangs together, which makes changing it for general consumption dicey. There also may be a degree of religion at work here, because if I can figure out how to do it from cold in a few hours the author(s) could crash this out a proper implementation in 5 minutes flat. So someone chooses not to.

It took some faffing about and I ended up with questions, I’m hi-jacking this thread because we don’t need to use a nagios type check script to do this…

Contrary to received wisdom snmp has the required information it needs a little messing about but it’s there. In fact the “system” graphs actually do show the right answer, but they’re buried away and again not as clear cut as I wanted.

Anyway - geek stuff. The graphs are constructed from mempool_free, mempool_total and mempool_used, these are in the table mempool. There are three rows per device, physical virtual and swap. They are not populated from where I expected

/opt/librenms/includes/polling/ucd-mib.inc.php

This collects all the good stuff and I spent time bending this to no avail, it gradually dawned on me that it has nothing to do with the graphs that I wanted to change. But it does have the values I really want inserted into the DB pulled in.

The next stop was mempools.inc.php this is in the same directory and seemed much more promising. It rapidly became apparent that my biggest single gripe was the value being inserted into mempool_used, but it wasn’t being set here…, yet I could prove it was, after some further debugging I realised mempools_in was calling mempools/hrstorage.inc.php. This is the one that sets up the values that get written to the DB. So that closes the circle if you want to change those graphs this is file that has to be altered.

So first question Why does LibreNMS collect all the snmp data using ucd_mib, then not actually write into the place where the most important graphs are drawn from?

Moving along and going into that file in a little more detail. This line defines mempool used and gets writte back to the DB

$mempool[‘used’] = ($entry[‘hrStorageUsed’] * $mempool[‘units’]);

Why?

It’s using HOST-RESOURCES-MIB rather than using ucd mib I wanted to use. I’m assuming there is a compelling reason, I’d love to know what it is.

Finally how can you make the graphs show real values?

For now I carved out the part of the ucb mib that collected the information I really want.

Put that at the top

$snmpdata = snmp_get_multi($device, [‘memTotalSwap.0’, ‘memAvailSwap.0’, ‘memTotalReal.0’, ‘memAvailReal.0’, ‘memTotalFree.0’, ‘memShared.0’, ‘memBuffer.0’, ‘memCached.0’], ‘-OQUs’, ‘UCD-SNMP-MIB’);

if (is_array($snmpdata[0])) {
[$memTotalSwap, $memAvailSwap, $memTotalReal, $memAvailReal, $memTotalFree, $memShared, $memBuffer, $memCached] = $snmpdata[0];
foreach (array_keys($snmpdata[0]) as $key) {
$$key = $snmpdata[0][$key];
}
}

$snmpdata = $snmpdata[0];

Then at the bottom added this

if ($mempool[‘mempool_descr’] == “Physical memory” )
{
$mempool[‘total’] = $memTotalReal * $mempool[‘units’];
$mempool[‘used’] = ($memTotalReal - ($memCached + $memBuffer + $memAvailReal))* $mempool[‘units’];
}
if ($mempool[‘mempool_descr’] == “Virtual memory” )
{
$mempool[‘total’] = ($memTotalReal + $memAvailSwap) * $mempool[‘units’];
$free_mem = ($memCached + $memBuffer + $memAvailReal + $memAvailSwap) * $mempool[‘units’];
$mempool[‘used’] = $mempool[‘total’] - $free_mem;
}
if ($mempool[‘mempool_descr’] == “Swap space” )
{
$mempool[‘total’] = $memTotalSwap * $mempool[‘units’];
$mempool[‘used’] = ($memTotalSwap - $memAvailSwap) * $mempool[‘units’];
}

This is pretty rubbish and achieves an end and am reluctant to spend any time on this without understanding why the HOSTRESOURCES mib is being used to populate that DB.

Not quite at the bottom must be before this line

$mempool[‘free’] = ($mempool[‘total’] - $mempool[‘used’]);

This gets fed back up and is used to calculate percentages and gets written to the DB.