@murrant - By Monday morning, everything was working correctly it appears. Whatever happened, was just happening during the time I was adding equipment and didn’t quite have the data right. Once the “fix” seems to have been applied, and I re-entered the user/pass/private then everything started to work like it was supposed to it seemed.
Thank you for the prompt update!
To all:
I added 3 devices between Friday and Sunday, and had to manually create the RRD folders for each device. Once that was done I still had an issue with the SNMPv3 credentials not working. I verified with snmpwalk that things were correct, but nothing I did in the GUI worked.
The fix listed above by @murrant, seems to have fixed my issues. I did re-enter the user information into the devices, and started getting graphs and data right away. This was well after manual creation of the RRD folders. I did not try to add another “net new” device, to see if it works wholesale now. I may do that later today/this week.
Thank you all!
Is your rrdcached server on a separate server from your pollers? This looks like it is working correctly, except there is no create command sent. If rrdtool version is not set correctly, is the only way the rrdcreate would be skipped there.
What is the output of lnms config:get rrdtool_version?
| Component | Version |
|---|---|
| LibreNMS | 23.9.1-71-g79b920243 (2023-10-10T13:46:07+02:00) |
| DB Schema | 2023_10_07_170735_increase_processes_cputime_length (261) |
| PHP | 8.1.11 |
| Python | 3.8.10 |
| Database | MariaDB 10.3.38-MariaDB-0ubuntu0.20.04.1 |
| RRDTool | 1.7.2 |
| SNMP | 5.8 |
| =========================================== |
[OK] Composer Version: 2.6.5
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK] Database and column collations are correct
[OK] Database schema correct
[OK] MySQl and PHP time match
[FAIL] Both Dispatcher Service and Python Wrapper were active recently, this could cause double polling
[OK] Dispatcher Service is enabled
[OK] Locks are functional
[OK] Python poller wrapper is polling
[OK] Redis is unavailable
[OK] rrdtool version ok
[OK] Connected to rrdcached
Creating the rrd folder started to populate the graphs.
This device is running snmpv2
My newly installed local testing VM server for LibreNMS did not encounter the issue. The problem only arose after installing and using rrdcached on the new server, which caused the rrd folder to stop being created.
It appears that turning off or not using rrdcached resolved the issue of the rrd folder not being automatically created, at least in my case. I tested this on my production LibreNMS server, and it can now create rrd folders when adding new devices.
I ran the following command on both servers after switching to the librenms user to disable rrdcached: lnms config:set rrdcached ""
Still don’t know the cause. I need more troubleshooting info.
I highly suspect this is related to using rrdcached and not having rrdtool_version set to a high enough version, but no one has confirmed or denied that guess.
The problem is in Poller.php file. Line 281:
if (Config::get('rrd.enable', true) && ! Config::get('rrdcached') && ! is_dir($host_rrd)) {
If you remove the condition about rrdcached it creates directories again. Don’t know why this condidion was added and why there is an expectation that rrdache will create subdirectories.
@estamico Thanks I was about to point that out for others to investigate!
Can you use dump() or dd() to figure out which of the checks is coming out to false?
@estamico wait, that is different than where I thought the problem is. rrdcached does create subdirectories with the -R option!
So the issue might be that LibreNMS now expects rrdcached to use -R but before it didn’t require it.
If this is the case, there are two options for fixing this:
so there must be something wrong with rrdcached or params sent to rrdcached. rrdcached is running with -R option on server with librenms
/usr/bin/rrdcached -B -F -R -w 1800 -z 1800 -t 4 -b /opt/librenms/rrd/ -j /var/lib/rrdcached/journal/ -G librenms -U librenms -p /var/run/rrdcached.pid -s librenms -l unix:/var/run/rrdcached.sock
On my server there is 1.7.0-1build1 version of rrdcached but I can see that @Andreas_Larsen has 1.7.2 and the problem is present too.
I see that some of this may be related to rrdcached. I do have this running and am at 1.7.2 and may still have the issue. I’ll get some more hardware added this week and update the thread.
Sorry for the delayed response. I wasn’t able to respond yesterday.
My situation is a single server installation (rrdcached is also on the same server).
lnms config:get rrdtool_version returns 1.7.2
Regarding the if-condition in Poller.php Line 281. The values returned from config are:
lnms config:get rrd.enable returns nothing
lnms config:get rrdcached returns unix:/var/run/rrdcached.sock
Please let me know if I still need to dump the variables from the script given this information.
Lastly in /etc/default/rrdcached rrdcached is started with the -R option.
Same problem here. It just suddenly stopped working but I don’t exactly know when as adding new devices isn’t a common occurrence any more, but somewhere in the last month or so.
rrdcached is running with -R, exactly according to the librenms rrdcached manual.
Single server setup.
the rrd.enable also returns nothing for me and rrdcached returns the correct unix socket.
===========================================
| Component | Version |
|---|---|
| LibreNMS | 23.9.1-81-ga5198d7d5 (2023-10-12T16:15:03+02:00) |
| DB Schema | 2023_10_07_231037_application_metrics_add_primary_key (262) |
| PHP | 8.1.24 |
| Python | 3.6.8 |
| Database | MariaDB 10.5.16-MariaDB |
| RRDTool | 1.7.0 |
| SNMP | 5.8 |
| =========================================== |
I’m having the same problem too. It’s with a new installation (two, actually).
It cocerns a single server installation, rrdcached is also on the same server.
lnms config:get rrdtool_version gives 1.7.0.
With regard to the if-condition in Poller.php Line 281, the values returned from my config are:
lnms config:get rrd.enable returns nothing
lnms config:get rrdcached returns unix:/run/rrdcached.sock
My rrdcached is started as follows
ExecStart=/usr/bin/rrdcached -w 1800 -z 1800 -f 3600 -s librenms -U librenms -G librenms -B -R -j /var/tmp -l unix:/run/rrdcached.sock -t 4 -F -b /opt/librenms/rrd/
So, with the -R parameter. I’m wondering what the idea is behind the rrd.enable; don’t we all want pretty graphs? 
(probably exists for testing code without creating graphs?)
Validation is ok.
===========================================
Component | Version
--------- | -------
LibreNMS | 23.9.1-90-g78a4bfb9d (2023-10-14T05:45:03+02:00)
DB Schema | 2023_10_12_184652_bgp_peers_table_unsigned_stats (265)
PHP | 8.1.24
Python | 3.6.8
Database | MariaDB 10.3.39-MariaDB
RRDTool | 1.7.0
SNMP | 5.8
===========================================
If any additional info is needed, I’d be happy to provide it.
Still trying to track down the cause.
It looks like rrdcached is outputing ERROR: realpath(snmpsim/poller-perf-applications.rrd): No such file or directory
When the checkRrdExists() function expects ERROR: snmpsim/poller-perf-applications.rrd: No such file or directory
If I fix the check to handle it, I still get an error during rrdcreate ERROR: realpath(snmpsim/poller-perf-applications.rrd): No such file or directory
Apparently, when you are running rrdtool and rrdcached on the same device for some reason it will not create sub directories. So we need to always make the directory on every poller to avoid this odd behavior.
I started at a new place and found a librenms instance running already, I added 2 new nodes and ran into this as well, had to manually create the node folders with librenms:librenms 740 owner/permissions to get it to start working. Hardest part was just getting someone who had sudo access to make the folders for me since I didn’t have ssh capabilities 
Sounds like a different problem. Run ./validate.php on that install.
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.