@murrant - By Monday morning, everything was working correctly it appears. Whatever happened, was just happening during the time I was adding equipment and didn’t quite have the data right. Once the “fix” seems to have been applied, and I re-entered the user/pass/private then everything started to work like it was supposed to it seemed.
I added 3 devices between Friday and Sunday, and had to manually create the RRD folders for each device. Once that was done I still had an issue with the SNMPv3 credentials not working. I verified with snmpwalk that things were correct, but nothing I did in the GUI worked.
The fix listed above by @murrant, seems to have fixed my issues. I did re-enter the user information into the devices, and started getting graphs and data right away. This was well after manual creation of the RRD folders. I did not try to add another “net new” device, to see if it works wholesale now. I may do that later today/this week.
Is your rrdcached server on a separate server from your pollers? This looks like it is working correctly, except there is no create command sent. If rrdtool version is not set correctly, is the only way the rrdcreate would be skipped there.
What is the output of lnms config:get rrdtool_version?
[OK] Composer Version: 2.6.5
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK] Database and column collations are correct
[OK] Database schema correct
[OK] MySQl and PHP time match
[FAIL] Both Dispatcher Service and Python Wrapper were active recently, this could cause double polling
[OK] Dispatcher Service is enabled
[OK] Locks are functional
[OK] Python poller wrapper is polling
[OK] Redis is unavailable
[OK] rrdtool version ok
[OK] Connected to rrdcached
Creating the rrd folder started to populate the graphs.
This device is running snmpv2
My newly installed local testing VM server for LibreNMS did not encounter the issue. The problem only arose after installing and using rrdcached on the new server, which caused the rrd folder to stop being created.
It appears that turning off or not using rrdcached resolved the issue of the rrd folder not being automatically created, at least in my case. I tested this on my production LibreNMS server, and it can now create rrd folders when adding new devices.
I ran the following command on both servers after switching to the librenms user to disable rrdcached: lnms config:set rrdcached ""
Still don’t know the cause. I need more troubleshooting info.
I highly suspect this is related to using rrdcached and not having rrdtool_version set to a high enough version, but no one has confirmed or denied that guess.
Apparently, when you are running rrdtool and rrdcached on the same device for some reason it will not create sub directories. So we need to always make the directory on every poller to avoid this odd behavior.
I started at a new place and found a librenms instance running already, I added 2 new nodes and ran into this as well, had to manually create the node folders with librenms:librenms 740 owner/permissions to get it to start working. Hardest part was just getting someone who had sudo access to make the folders for me since I didn’t have ssh capabilities