Total Device Traffic error drawing graph

[bwilli37]

Since yesterday, devices with a large number of ports (e.g. large switch stacks, Cisco VSS, Cisco FEX) are failing to draw the Total Device Traffic graph that appears on the device overview page. This happens for a variety of device and OS types.

Everything looks good in validate.php, LibreNMS code is up to date, no errors are seen when running the rrdtool command from the CLI (it outputs what other threads have lead me to believe is the graph image). There is no indication of any errors when viewing the RRD Tool Output in the web UI. I’ve also restarted rrdcached, httpd, mariadb and the entire server but no difference.

Any help is appreciated. These graphs are handy. Thanks!

====================================
Component | Version
--------- | -------
LibreNMS  | 21.4.0-62-ge40e1069d
DB Schema | 2021_04_08_151101_add_foreign_keys_to_port_group_port_table (208)
PHP       | 7.3.20
Python    | 3.6.8
MySQL     | 10.3.27-MariaDB
RRDTool   | 1.7.0
SNMP      | NET-SNMP 5.8
====================================

[OK]    Composer Version: 2.0.11
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct

Device overview page:

Clicking the Total Device Traffic graph then show RRD Command output at the bottom, no errors:

[djuneardy]

me too , total device traffic on cumulus network error drawing

[Pr_M]

Same here. I’ve got several devices with draw error after upgrade.

[SMU_Instructors]

I came here to find this post. Got the same issue. Any fix yet?

I found it happens with a recent update to LibreNMS. I have a backup of the previous day, and rolled back to it. Graphs showed up fine. Then, the following day, graphs are back to “Draw Error”. Only happens for switch stacks with many ports. but not all of them. And only for the “Overall Traffic” graph.

[mac_prod]

hello,
I have the same problem of displaying the “overall” graph on a device. I checked all the RRD files (366…) and all the individual graphs are there.
Before this post, I of course did a “./validate.php” and a “./validate.php -g rrdcheck”, which allowed me to verify that about 60 of our thousands of RRDs were damaged.
I deleted them and that fixed all the individual graph problems, except the overall one…

[Colin_A]

I have the same exact issue - devices with large numbers of interfaces not displaying Overall Traffic graphs. Validate is good and running the ./validate.php -g rrdcheck command did not find any damaged files, so @mac_prod’s suggestion didn’t work for me.
Anyone have any other suggestions?

[PipoCanaja]

Could you try this command on your LibreNMS server to find out the maximum length of a shell command:
getconf ARG_MAX
On my machine, it would take about 2000 interfaces to hit the getconf ARG_MAX limit, which is 2MBytes. Each interface takes about 1kByte of space in the rrdtool command.

[samotelf]

I have this in all my HP 5412 switches after updating too.
But only in the traffic graph.

This is my output:
-bash-4.2$ ./validate.php -g rrdcheck

Component	Version
LibreNMS	21.4.0-84-g500b0ac
DB Schema	2021_04_08_151101_add_foreign_keys_to_port_group_port_table (208)
PHP	7.3.19
Python	3.6.8
MySQL	10.5.9-MariaDB
RRDTool	1.4.8
SNMP	NET-SNMP 5.7.2

====================================

[OK] Composer Version: 2.0.13
[OK] Dependencies up-to-date.
Checking rrdcheck:
Scanning 13237 rrd files in /opt/librenms/rrd…
Status: 13237/13237 - Complete
OK

Output of getconf ARG_MAX:
-bash-4.2$ getconf ARG_MAX
2097152
-bash-4.2$

Please kindly help.

[murrant]

@PipoCanaja is right.

Can some of you test https://github.com/librenms/librenms/pull/12868 and give feedback?

(legends will be off, don’t worry about that)

[brockcj]

Seems to resolve it for me. I was also seeing broken poller performance graphs as well showing the same error and those are also working after applying 12868

[bwilli37]

12868 fixed it for me as well. Thank you!

Other outputs for reference:

-bash-4.2$ getconf ARG_MAX
2097152

-bash-4.2$ ./validate.php -g rrdcheck
====================================
Component | Version
--------- | -------
LibreNMS  | 21.4.0-84-g500b0ac6f
DB Schema | 2021_04_08_151101_add_foreign_keys_to_port_group_port_table (208)
PHP       | 7.3.20
Python    | 3.6.8
MySQL     | 10.3.27-MariaDB
RRDTool   | 1.7.0
SNMP      | NET-SNMP 5.8
====================================

[OK]    Composer Version: 2.0.11
[OK]    Dependencies up-to-date.
Checking rrdcheck:
Scanning 42161 rrd files in /opt/librenms/rrd...
Status: 42161/42161 - Complete
 OK

[mac_prod]

hello,
without any additional modification, the display of the overall is now normal (after two days of waiting)! Change on librenms with cron daily.sh?

[PipoCanaja]

Yes @mac_prod. Update takes place automatically during daily.sh (unless you run stable version, which updates only once a month). So this is expected.
Bye