RRD Folders not getting created

Help
1

Hello team:

I’m having a new issue with an existing installation. I recently moved from CentOS to Ubuntu on a new VMware host, but don’t think that’s it related as I have added devices after the migration without issue.

This weekend though, I’ve added 4 different devices/sites which are all Meraki. I added 3 MX devices, and the cloud controller. For all 4 additions, the RRD folders are not getting created during the install, so I’m getting a “realpath blah blah blah” message in red instead of the graphs.

To get things “working” I had to manually create the folder under the RRD folder, then graphs appears. BUT for 2 of the devices, I have changed the SNMP credentials in the GUI (latest, validation passed), but don’t think they’re saving as I still don’t have any data, and when hitting “save” nothing really happens. No indication that anything saved, just a quick flash of the screen.

What should I be looking at to troubleshoot this? I’m at a loss as the permissions seem correct, and validate shows no issues what so ever.

Thanks!

1 Like
2

Can you please poller debug output. lnms device:poll -vv <hostname> (of a device that doesn’t have a folder yet)

3

Is there a particular section your are interested in seeing? The output is pretty lengthy. I am seeing a number of invalid SNMP lines such as:
Invalid snmp_walk() data = ifEntry = No Such Object available on this agent at this OID

I did manually create the RRD folders, so I’m getting some graphing off a couple devices. It appears though that the SNMP credential changes I’m making in LibreNMS are not “taking” or saving. Is there a sport to review that?

Thank you!

4

Fix bug saving snmpv3 device changes by murrant · Pull Request #15419 · librenms/librenms · GitHub might help

5

Good Day Everyone!

We are also encountering this problem… for newly devices added, it doesn’t create new rrd folders on “opt/librenms/rrd” which I think was the cause for the “ERROR: realpath…”

After running “lnms device:poll -vvv (device-ip) -m os” under librenms user, the errors I see are only

“ERROR: realpath((device_ip)/poller-perf-core.rrd): No such file or directory” and other *.rrd files

I also re-did the things here RRDCached - LibreNMS Docs, still negative

System:
LibreNMS | 23.9.1-70-ga6239a733 (2023-10-09T08:00:05+08:00)
DB Schema | 2023_10_07_170735_increase_processes_cputime_length (261)
PHP | 8.1.24
Python | 3.8.10
Database | MariaDB 10.5.11-MariaDB-1:10.5.11+maria~bionic
RRDTool | 1.7.2
SNMP | 5.8

6

I’m also facing this issue for new devices only. Existing devices are OK and still have their graphs updated. I’m also using RRDCached, but since existing devices still work I’m not leaning towards this being the issue.

Output of lnms device:poll -vvv <FQDN-OF-DEVICE> for a newly added device:

librenms@<redacted>:~$ lnms device:poll -vvv <FQDN-OF-DEVICE>
Override poller modules: core, unix-agent, os, availability, ipmi, sensors, processors, mempools, storage, netstats, hr-mib, ucd-mib, ipSystemStats, ports, xdsl, customoid, bgp-peers, junose-atm-vp, printer-supplies, ucd-diskio, vminfo, wifi, wireless, ospf, isis, cisco-ipsec-flow-monitor, cisco-remote-access-monitor, cisco-cef, slas, cisco-mac-accounting, cipsec-tunnels, cisco-ace-loadbalancer, cisco-ace-serverfarms, cisco-asa-firewall, cisco-voice, cisco-cbqos, cisco-otv, cisco-qfp, cisco-vpdn, nac, netscaler-vsvr, aruba-controller, entity-physical, entity-state, applications, stp, ntp, loadbalancers, mef, mpls  
SQL[select `migration` from `migrations` order by `id` desc limit 1 [] 0.49ms] 
  
SQL[select count(*) as aggregate from `migrations` [] 0.29ms] 
 
SQL[select version() [] 0.19ms] 
  
===========================================
Component | Version
--------- | -------
LibreNMS  | 23.9.1-70-ga6239a733 (2023-10-09T02:00:05+02:00)
DB Schema | 2023_10_07_170735_increase_processes_cputime_length (261)
PHP       | 8.1.24
Python    | 3.8.10
Database  | MariaDB 10.3.38-MariaDB-0ubuntu0.20.04.1
RRDTool   | 1.7.2
SNMP      | 5.8
===========================================
  
Updating os_def.cache  
Starting polling run:
  
SQL[select `device_id` from `devices` where `hostname` = ? ["<FQDN-OF-DEVICE>"] 0.65ms] 
  
SQL[select * from `devices` where `device_id` = ? limit 1 [252] 0.44ms] 
  
Hostname:  <FQDN-OF-DEVICE> 
ID:        252
OS:        ibm-i
IP:        <IP-OF-DEVICE>
  
Attempting to initialize OS: ibm-i  
OS initialized as Generic (ibm-i)  
SQL[select * from `devices_attribs` where `devices_attribs`.`device_id` = ? and `devices_attribs`.`device_id` is not null [252] 0.29ms] 
  
[FPING] '/usr/bin/fping' '-e' '-q' '-c' '3' '-p' '500' '-t' '500' '-O' '0' '<FQDN-OF-DEVICE>'
  
response: xmt/rcv/%loss = 3/3/0%, min/avg/max = 4.53/4.65/4.84  
SNMP['/usr/bin/snmpget' '-M' '/opt/librenms/mibs' '-m' 'SNMPv2-TC:SNMPv2-MIB:IF-MIB:IP-MIB:TCP-MIB:UDP-MIB:NET-SNMP-VACM-MIB' '-v1' '-c' 'PRLGGRABRO' '-OQXUte' '-t' '5' 'udp:<FQDN-OF-DEVICE>:161' 'SNMPv2-MIB::sysObjectID.0']  
SNMPv2-MIB::sysObjectID.0 = SNMPv2-SMI::enterprises.2.6.11
  
  
SQL[insert into `device_perf` (`min`, `max`, `avg`, `xmt`, `rcv`, `loss`, `debug`, `device_id`, `timestamp`) values (?, ?, ?, ?, ?, ?, ?, ?, ?) [4.53,4.84,4.65,3,3,0,"{\"poller_name\":\"prlg-mon-01\"}",252,"2023-10-10 09:27:59"] 3.09ms] 
  
SQL[update `devices` set `last_ping_timetaken` = ? where `device_id` = ? [4.65,252] 2.7ms] 
  
RRD[last <FQDN-OF-DEVICE>/ping-perf.rrd  --daemon unix:/var/run/rrdcached.sock]  
RRDtool Output: -1
ERROR: realpath(<FQDN-OF-DEVICE>/ping-perf.rrd): No such file or directory
RRD[update <FQDN-OF-DEVICE>/ping-perf.rrd N:4.65 --daemon unix:/var/run/rrdcached.sock]  
RRDtool Output: SQL[select * from `device_outages` where `device_outages`.`device_id` = ? and `device_outages`.`device_id` is not null and `up_again` is null order by `going_down` desc limit 1 [252] 0.23ms] 
  
#### Load poller module core ####
  
Module enabled: Global + | OS   | Device   | Manual    
SNMP['/usr/bin/snmpget' '-M' '/opt/librenms/mibs' '-m' 'SNMPv2-TC:SNMPv2-MIB:IF-MIB:IP-MIB:TCP-MIB:UDP-MIB:NET-SNMP-VACM-MIB' '-v1' '-c' 'PRLGGRABRO' '-OQXUte' '-On' '-t' '5' 'udp:<FQDN-OF-DEVICE>:161' 'SNMPv2-MIB::sysUpTime.0']  
.1.3.6.1.2.1.1.3.0 = 6972603
  
  
SNMP['/usr/bin/snmpget' '-M' '/opt/librenms/mibs' '-m' 'SNMPv2-TC:SNMPv2-MIB:IF-MIB:IP-MIB:TCP-MIB:UDP-MIB:NET-SNMP-VACM-MIB' '-v1' '-c' 'PRLGGRABRO' '-OQXUte' '-t' '5' 'udp:<FQDN-OF-DEVICE>:161' 'SNMP-FRAMEWORK-MIB::snmpEngineTime.0' 'HOST-RESOURCES-MIB::hrSystemUptime.0']  
Exitcode: 2 ["Error in packet\nReason: (noSuchName) There is no such variable name in this MIB.\nFailed object: SNMP-FRAMEWORK-MIB::snmpEngineTime.0\n\n"] 
HOST-RESOURCES-MIB::hrSystemUptime.0 = 550744598
  
Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: SNMP-FRAMEWORK-MIB::snmpEngineTime.0

  
Uptime seconds: 5507446
  
RRD[last <FQDN-OF-DEVICE>/uptime.rrd  --daemon unix:/var/run/rrdcached.sock]  
RRDtool Output: -1
ERROR: realpath(<FQDN-OF-DEVICE>/uptime.rrd): No such file or directory
RRD[update <FQDN-OF-DEVICE>/uptime.rrd N:5507446 --daemon unix:/var/run/rrdcached.sock]  
RRDtool Output: ERROR: realpath(<FQDN-OF-DEVICE>/ping-perf.rrd): No such file or directory
Uptime: 2 months 1 week 17 hours 50 minutes 46 seconds  
SQL[update `devices` set `uptime` = ? where `device_id` = ? [5507446,252] 2.85ms] 
  
  
>> SNMP: [3/0.06s] MySQL: [5/0.09s] RRD: [4/0.00s]  
>> Runtime for poller module 'core': 0.0459 seconds with 52648 bytes  
RRD[last <FQDN-OF-DEVICE>/poller-perf-core.rrd  --daemon unix:/var/run/rrdcached.sock]  
RRDtool Output: -1
ERROR: realpath(<FQDN-OF-DEVICE>/poller-perf-core.rrd): No such file or directory
RRD[update <FQDN-OF-DEVICE>/poller-perf-core.rrd N:0.045899152755737 --daemon unix:/var/run/rrdcached.sock]  
RRDtool Output: ERROR: realpath(<FQDN-OF-DEVICE>/uptime.rrd): No such file or directory
#### Unload poller module core ####

./validate.php and Web-UI validate are showing no errors:

librenms@<redacted>:~$ ./validate.php 
===========================================
Component | Version
--------- | -------
LibreNMS  | 23.9.1-70-ga6239a733 (2023-10-09T02:00:05+02:00)
DB Schema | 2023_10_07_170735_increase_processes_cputime_length (261)
PHP       | 8.1.24
Python    | 3.8.10
Database  | MariaDB 10.3.38-MariaDB-0ubuntu0.20.04.1
RRDTool   | 1.7.2
SNMP      | 5.8
===========================================

[OK]    Composer Version: 2.6.5
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    Database and column collations are correct
[OK]    Database schema correct
[OK]    MySQl and PHP time match
[OK]    Active pollers found
[OK]    Dispatcher Service not detected
[OK]    Locks are functional
[OK]    Python poller wrapper is polling
[OK]    Redis is unavailable
[OK]    rrdtool version ok
[OK]    Connected to rrdcached
7

We are facing the same issue.
If we create the RRD folder manually - prior to adding the device “it works”.
I hope this may help in understanding what’s wrong.

our validate.php:

===========================================

Component Version
LibreNMS 23.9.1-70-ga6239a733 (2023-10-09T00:00:05+00:00)
DB Schema 2023_10_07_170735_increase_processes_cputime_length (261)
PHP 8.1.9
Python 3.8.10
Database MariaDB 10.3.38-MariaDB-0ubuntu0.20.04.1
RRDTool 1.7.2
SNMP 5.8
===========================================

[OK] Composer Version: 2.6.4
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK] Database and column collations are correct
[OK] Database schema correct
[OK] MySQl and PHP time match
[OK] Active pollers found
[OK] Dispatcher Service is enabled
[OK] Locks are functional
[OK] Python wrapper cron entry is not present
[OK] Redis is unavailable
[OK] rrdtool version ok
[OK] Connected to rrdcached

8

Same issue here as of a few days ago. Also snmpV3.

9

We are also running into the same issue with all new devices being created as of a few days ago. All devices are running SNMPv3 and the output of lnms device:poll <hostname> appears to show no errors.

Once I’ve created the directory under /opt/librenms/rrd/<hostname_or_ip>/ the rrd files start populating for the device.

For completeness, here’s the LibreNMS/RRD versions I’m running:

| Version |  23.9.1 - Tue Sep 19 2023 19:15:05 GMT-0500|
| RRDtool |  1.7.2                                     |
1 Like
10

@murrant - By Monday morning, everything was working correctly it appears. Whatever happened, was just happening during the time I was adding equipment and didn’t quite have the data right. Once the “fix” seems to have been applied, and I re-entered the user/pass/private then everything started to work like it was supposed to it seemed.

Thank you for the prompt update!

11

To all:

I added 3 devices between Friday and Sunday, and had to manually create the RRD folders for each device. Once that was done I still had an issue with the SNMPv3 credentials not working. I verified with snmpwalk that things were correct, but nothing I did in the GUI worked.

The fix listed above by @murrant, seems to have fixed my issues. I did re-enter the user information into the devices, and started getting graphs and data right away. This was well after manual creation of the RRD folders. I did not try to add another “net new” device, to see if it works wholesale now. I may do that later today/this week.

Thank you all!

12

Is your rrdcached server on a separate server from your pollers? This looks like it is working correctly, except there is no create command sent. If rrdtool version is not set correctly, is the only way the rrdcreate would be skipped there.

What is the output of lnms config:get rrdtool_version?

13

I have the same issue but for a newly created device
./validate.php

Component Version
LibreNMS 23.9.1-71-g79b920243 (2023-10-10T13:46:07+02:00)
DB Schema 2023_10_07_170735_increase_processes_cputime_length (261)
PHP 8.1.11
Python 3.8.10
Database MariaDB 10.3.38-MariaDB-0ubuntu0.20.04.1
RRDTool 1.7.2
SNMP 5.8
===========================================

[OK] Composer Version: 2.6.5
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK] Database and column collations are correct
[OK] Database schema correct
[OK] MySQl and PHP time match
[FAIL] Both Dispatcher Service and Python Wrapper were active recently, this could cause double polling
[OK] Dispatcher Service is enabled
[OK] Locks are functional
[OK] Python poller wrapper is polling
[OK] Redis is unavailable
[OK] rrdtool version ok
[OK] Connected to rrdcached

Creating the rrd folder started to populate the graphs.
This device is running snmpv2

14

My newly installed local testing VM server for LibreNMS did not encounter the issue. The problem only arose after installing and using rrdcached on the new server, which caused the rrd folder to stop being created.

It appears that turning off or not using rrdcached resolved the issue of the rrd folder not being automatically created, at least in my case. I tested this on my production LibreNMS server, and it can now create rrd folders when adding new devices.

I ran the following command on both servers after switching to the librenms user to disable rrdcached: lnms config:set rrdcached ""

15

Still don’t know the cause. I need more troubleshooting info.
I highly suspect this is related to using rrdcached and not having rrdtool_version set to a high enough version, but no one has confirmed or denied that guess.

16

The problem is in Poller.php file. Line 281:

if (Config::get('rrd.enable', true) && ! Config::get('rrdcached') && ! is_dir($host_rrd)) {

If you remove the condition about rrdcached it creates directories again. Don’t know why this condidion was added and why there is an expectation that rrdache will create subdirectories.

17

@estamico Thanks I was about to point that out for others to investigate!

Can you use dump() or dd() to figure out which of the checks is coming out to false?

18

@estamico wait, that is different than where I thought the problem is. rrdcached does create subdirectories with the -R option!

So the issue might be that LibreNMS now expects rrdcached to use -R but before it didn’t require it.

19

If this is the case, there are two options for fixing this:

  1. Always create the directory. This will create useless local directories for those with remote rrdcached. but not break anyone.
  2. Add a validate check to make sure rrdcached is running with -R.
20

so there must be something wrong with rrdcached or params sent to rrdcached. rrdcached is running with -R option on server with librenms

/usr/bin/rrdcached -B -F -R -w 1800 -z 1800 -t 4 -b /opt/librenms/rrd/ -j /var/lib/rrdcached/journal/ -G librenms -U librenms -p /var/run/rrdcached.pid -s librenms -l unix:/var/run/rrdcached.sock

On my server there is 1.7.0-1build1 version of rrdcached but I can see that @Andreas_Larsen has 1.7.2 and the problem is present too.