Latency graphs not showing for new ping-only devices

tievolu · 19 April 2024 17:06

I’m having issues adding new ping-only devices - the devices are added just fine, and everything works apart from the latency graphs. These two files do not get created and the web UI shows an error:

ping-perf.rrd
icmp-perf.rrd

This only affects new devices. My pre-existing ping-only devices are working fine, graphs and all.

I don’t know when this problem was introduced - I haven’t added a ping-only device for a long time before today. I did notice this recent change in the github log though, and thought it might be related.

I can reproduce this easily, by simply adding a new ping-only device to any reachable address, e.g. 1.1.1.1.

tievolu · 19 April 2024 17:08

Example output from lnms device:poll with option -vvv:

RRD[create /opt/librenms/rrd/1.1.1.1/icmp-perf.rrd --step 300 --source /opt/librenms/rrd/1.1.1.1/ping-perf.rrd DS:avg=ping[1]:GAUGE:600:0:65535 DS:xmt:GAUGE:600:0:65535 DS:rcv:GAUGE:600:0:65535 DS:min:GAUGE:600:0:65535 DS:max:GAUGE:600:0:65535   RRA:AVERAGE:0.5:1:2016 RRA:AVERAGE:0.5:6:1440 RRA:AVERAGE:0.5:24:1440 RRA:AVERAGE:0.5:288:1440 RRA:MIN:0.5:1:2016 RRA:MIN:0.5:6:1440 RRA:MIN:0.5:24:1440 RRA:MIN:0.5:288:1440 RRA:MAX:0.5:1:2016 RRA:MAX:0.5:6:1440 RRA:MAX:0.5:24:1440 RRA:MAX:0.5:288:1440 RRA:LAST:0.5:1:2016  -O]
RRDtool Output: ERROR: error checking for source RRD /opt/librenms/rrd/1.1.1.1/ping-perf.rrd: No such file or directory
RRD[update /opt/librenms/rrd/1.1.1.1/icmp-perf.rrd N:8.83:3:3:8.79:8.88]
RRDtool Output: ERROR: opening '/opt/librenms/rrd/1.1.1.1/icmp-perf.rrd': No such file or directory

tievolu · 19 April 2024 17:17

Output from validate.php:

===========================================
Component | Version
--------- | -------
LibreNMS  | 24.3.0-26-g36d14480f (2024-04-19T16:22:06+01:00)
DB Schema | 2024_04_10_093513_remove_device_perf (291)
PHP       | 8.1.11
Python    | 3.7.3
Database  | MariaDB 10.3.36-MariaDB-0+deb10u2
RRDTool   | 1.7.1
SNMP      | 5.7.3
===========================================

[OK]    Composer Version: 2.7.2
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    Database and column collations are correct
[OK]    Database schema correct
[OK]    MySQL and PHP time match
[OK]    Active pollers found
[OK]    Dispatcher Service not detected
[OK]    Locks are functional
[OK]    Python poller wrapper is polling
[OK]    Redis is unavailable
[OK]    rrd_dir is writable
[OK]    rrdtool version ok

murrant · 19 April 2024 19:27

Looks like it is trying to migrate data from ping-perf.rrd even when it doesn’t exist.

Needs a bug fix.

murrant · 20 April 2024 00:54

github.com/librenms/librenms

Skip rrd sources that do not exist

librenms:master ← murrant:fix_rrd_source

opened 08:10PM - 19 Apr 24 UTC

murrant

+24 -9

https://community.librenms.org/t/latency-graphs-not-showing-for-new-ping-only-de…vices/24150 #### Please note > Please read this information carefully. You can run `./lnms dev:check` to check your code before submitting. - [x] Have you followed our [code guidelines?](https://docs.librenms.org/Developing/Code-Guidelines/) - [x] If my Pull Request does some changes/fixes/enhancements in the WebUI, I have inserted a screenshot of it. - [x] If my Pull Request makes discovery/polling/yaml changes, I have added/updated [test data](https://docs.librenms.org/Developing/os/Test-Units/). #### Testers If you would like to test this pull request then please run: `./scripts/github-apply <pr_id>`, i.e `./scripts/github-apply 5926` After you are done testing, you can remove the changes with `./scripts/github-remove`. If there are schema changes, you can ask on discord how to revert.

tievolu · 20 April 2024 10:56

Yep, that fixed it. Thanks!

warrenashcroft · 23 April 2024 18:14

This fix doesn’t work where rrdcached is being used, it’s checking the existence of a local file.

Despite the file not existing, it still errors

RRD[last 8.8.8.8/icmp-perf.rrd  --daemon 172.24.200.5:42217]  
ERROR: [email protected]:42217: RRD Error: opening '/data/librenms/rrd/8.8.8.8/icmp-perf.rrd': No such file or directory
RRD[last 8.8.8.8/ping-perf.rrd  --daemon 172.24.200.5:42217]  
RRD[create 8.8.8.8/icmp-perf.rrd --step 300 --source 8.8.8.8/ping-perf.rrd DS:avg=ping[1]:GAUGE:600:0:65535 DS:xmt:GAUGE:600:0:65535 DS:rcv:GAUGE:600:0
:65535 DS:min:GAUGE:600:0:65535 DS:max:GAUGE:600:0:65535   RRA:AVERAGE:0.5:1:2016 RRA:AVERAGE:0.5:6:1440 RRA:AVERAGE:0.5:24:1440 RRA:AVERAGE:0.5:288:1440 RRA:MIN:0.5:1
:2016 RRA:MIN:0.5:6:1440 RRA:MIN:0.5:24:1440 RRA:MIN:0.5:288:1440 RRA:MAX:0.5:1:2016 RRA:MAX:0.5:6:1440 RRA:MAX:0.5:24:1440 RRA:MAX:0.5:288:1440 RRA:LAST:0.5:1:2016  -
O --daemon 172.24.200.5:42217]  
RRDtool Output: RRD[update 8.8.8.8/icmp-perf.rrd N:1.71:6:6:1.66:1.9 --daemon 172.24.200.5:42217]  
RRDtool Output: ERROR: error checking for source RRD 8.8.8.8/ping-perf.rrd: No such file or directory
RRDtool Output: ERROR: [email protected]:42217: No such file: /data/librenms/rrd/8.8.8.8/icmp-perf.rrd

murrant · 24 April 2024 02:00

I noticed that, the only solution I have for now is to disable data migration when the poller doesn’t have direct file access.

warrenashcroft · 25 April 2024 02:52

Would it be advisable to generate bulk rrd migration commands to run manually on the rrdcached server?

As it stands, the latest major changes to ping graphing have completely broken all installs using rrdcached. Migration isn’t happening, polled data is being lost and ping graphs aren’t displaying because icmp-perf.rrd is never getting created.

Tony_Silva · 25 April 2024 09:33

I also have the problem even after the fix was applied. Attaching a screenshot if that will help.

I tried to add a new device to be monitored with ping only and in the overview page where ICMP graph should be, it shows this error:
Error Drawing Graph: ERROR: unknown option ‘–left-axis-format’

So it seems the bug is still affecting devices that were being monitored before and newly added devices.

dennypage · 26 April 2024 21:45

I second this request.

Tony_Silva · 30 April 2024 06:29

Any idea if this can be fixed?

MrNiHao · 2 May 2024 22:08

I had the same problem, but after ugrading rrdtool it was fixed.

Which version of rrdtool do you have? If you are running Red Hat/CentOS/Linux 7 then it would be 1.4.8. Upgrade to a higher version like 1.7 or 1.8 and it should be fixed after the poller has run.

dennypage · 3 May 2024 03:13

I was/am running the latest version of rrdtool (1.8.0) and rrdcached.

In order to get mine fixed, I had to shutdown rrdcached, copy all the rrd files locally to librenms, run librenms for a few polling cycles, and then shut everything down again and put the files back with rrdcached. A real pain, but it works.