RRD file locations with dispatcher

Help
#1

When asking for help and support, please provide as much information as possible. This should include:

  • Steps to reproduce an issue.

RRD files look to be updating on dispatcher and main server. I have run out of disk on dispatcher and wondering if I have something misconfigured instead. Otherwise, I guess I will have to give it more disk.

  • The output of ./validate.php

From main server:

[[email protected] ~]$ ./validate.php
===========================================
Component | Version
--------- | -------
LibreNMS  | 23.2.0 (2023-02-23T17:27:42-05:00)
DB Schema | 2022_08_15_084507_add_rrd_type_to_wireless_sensors_table (248)
PHP       | 8.1.10
Python    | 3.6.8
Database  | MariaDB 10.3.35-MariaDB
RRDTool   | 1.7.0
SNMP      | 5.8
===========================================

[OK]    Composer Version: 2.5.4
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    Database and column collations are correct
[OK]    Database schema correct
[OK]    MySQl and PHP time match
[OK]    Distributed Polling setting is enabled globally
[OK]    Connected to rrdcached
[OK]    Active pollers found
[OK]    Dispatcher Service is enabled
[OK]    Locks are functional
[OK]    No active python wrapper pollers found
[OK]    Redis is functional
[OK]    rrdtool version ok
[OK]    Connected to rrdcached

[[email protected] ~]$ ./validate.php
===========================================
Component | Version
--------- | -------
LibreNMS  | 23.2.0 (2023-02-23T17:27:42-05:00)
DB Schema | 2022_08_15_084507_add_rrd_type_to_wireless_sensors_table (248)
PHP       | 8.1.10
Python    | 3.6.8
Database  | MariaDB 10.3.35-MariaDB
RRDTool   | 1.7.0
SNMP      | 5.8
===========================================

[OK]    Composer Version: 2.5.4
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    Database and column collations are correct
[OK]    Database schema correct
[OK]    MySQl and PHP time match
[WARN]  Disk space where /opt/librenms/rrd is located is less than 512Mb
[OK]    Distributed Polling setting is enabled globally
[OK]    Connected to rrdcached
[OK]    Active pollers found
[OK]    Dispatcher Service is enabled
[OK]    Locks are functional
[OK]    No active python wrapper pollers found
[OK]    Redis is functional
[OK]    rrdtool version ok
[OK]    Connected to rrdcached

If it’s an issue with the WebUI then please consider including a screenshot and the browser version you are using.

n/a

If you are having troubles with discovery/polling include the pastebin output of:

./discovery.php -h HOSTNAME -d | ./pbin.sh
./poller.php -h HOSTNAME -r -f -d | ./pbin.sh

n/a

If you need to post any text longer than a few lines, please use a pastebin service such as https://p.libren.ms using non-expiring pastes.

#2

Note: this servers have been up for months

Just to illustrate the number of files changed over the course of the last day:

[[email protected] rrd]$ find . -mtime -1 | wc -l
2813

[[email protected] rrd]$ find . -mtime -1 | wc -l
33143
#3

Have a look at some of the cleanup options in Cleanup Options - LibreNMS Docs

$config['rrd_purge'] might be useful if you are happy to have old/removed ports/data cleaned up etc.

Otherwise if you don’t have enough disk space attempts to create new RRDs for new ports will fail.

#4

@pjchilds Thank you for the documentation help.

I was able to move the files away to a temporary disk to fix the immediately problem.

I did find the source of the problem. It looks like when the master service goes down while the dispatcher is up, the dispatcher will write locally till it is restarted. I happen to restart the main server process and the dispatcher disk didnt get bigger.

Do you know if there is anything in the documentation about this? I could write a cron script that checks the dispatcher’s RRD directory and alert me if there are any files there but there has to be something more elegant.