Error Drawing Graph (For select items)

Howdy.

Here’s an environment breakdown.

Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase                        OK
Updating Composer packages                         OK
Updating SQL-Schema                                OK
Updating submodules                                OK
Cleaning up DB                                     OK
Fetching notifications                             OK
Caching PeeringDB data                             OK

When running validate.php, the following output is seen:

====================================
Component | Version
--------- | -------
LibreNMS  | 1.43-127-g54de884
DB Schema | 268
PHP       | 7.0.32-0ubuntu0.16.04.1
MySQL     | 10.0.36-MariaDB-0ubuntu0.16.04.1
RRDTool   | 1.5.5
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.7.2
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct

From looking at older posts, I will stress that my RRDTool version is 1.5.5, not 1.6.x or 1.4.x.

The Problem

My specific issue is related to a processor usage of a single device (as far as I can tell).

image

Attached is a screenshot of an example of what I’m seeing. The processor usage graph is all red and when selected I’m given the “Error Drawing Graph”.

I read from another thread that I can append the image link’s address with ‘&debug=true’ to get an error of a sorts. That is attached here.

SQL[SELECT `devices`.*, `lat`, `lng` FROM `devices` LEFT JOIN locations ON `devices`.`location`=`locations`.`location` WHERE `device_id` = ? ["18"] 0.35ms] SQL[SELECT * FROM devices_attribs WHERE `device_id` = ? [18] 0.17ms] SQL[SELECT * FROM `vrf_lite_cisco` WHERE `device_id` = ? ["18"] 0.17ms] SQL[SELECT attrib_value FROM devices_attribs WHERE `device_id` = ? AND `attrib_type` = ? [18,"poll_mib"] 0.17ms] SQL[SELECT * FROM `processors` where `device_id` = ? [18] 0.16ms] 
Warning: substr_count(): Length value 14 exceeds string length in /opt/librenms/includes/rrdtool.inc.php on line 284

Warning: Invalid argument supplied for foreach() in /opt/librenms/html/includes/graphs/generic_multi_simplex_seperated.inc.php on line 23
graph /tmp/aMJGPcBmKkpgTykS -l 0 -u 100 -E --start 1538428800 --end 1538515200 --width 1512 --height 313 -c BACK#EEEEEE00 -c SHADEA#EEEEEE00 -c SHADEB#EEEEEE00 -c FONT#000000 -c CANVAS#FFFFFF00 -c GRID#a5a5a5 -c MGRID#FF9999 -c FRAME#5e5e5e -c ARROW#5e5e5e -R normal --font LEGEND:8:DejaVuSansMono --font AXIS:7:DejaVuSansMono --font-render-mode normal COMMENT:'Load % Now Min Max Avg\l'

command returned (OK u:0.00 s:0.00 r:0.00 )

graph /tmp/aMJGPcBmKkpgTykS --alt-autoscale-max --rigid -E --start 1538428800 --end 1538515200 --width 1512 --height 313 -c BACK#EEEEEE00 -c SHADEA#EEEEEE00 -c SHADEB#EEEEEE00 -c FONT#000000 -c CANVAS#FFFFFF00 -c GRID#a5a5a5 -c MGRID#FF9999 -c FRAME#5e5e5e -c ARROW#5e5e5e -R normal --font LEGEND:8:DejaVuSansMono --font AXIS:7:DejaVuSansMono --font-render-mode normal HRULE:0#555555 --title='Error Drawing Graph'

command returned (1593x372 OK u:0.06 s:0.00 r:0.06 )


Runtime 0.087s
MySQL [5/0.00s]: Cell[0/0.00s] Row[2/0.00s] Rows[-1/-0.00s] Column[4/0.00s] Update[0/0.00s] Insert[0/0.00s] Delete[0/0.00s]

When did this problem start

I was trying to setup SMART (HDD) monitoring on a server I was monitoring and I ran into a NaN graph problem. Part of my troubleshooting method was to check ./validate.php and I returned/executed the following permission-related commands.

[FAIL]  We have found some files that are owned by a different user than librenms, this will stop you updating automatically and / or rrd files being updated causing graphs to fail.
    [FIX] sudo chown -R librenms:librenms /opt/librenms
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/

Since then, this has been a problem.

Can anyone please shed some light on this?

Note: I’m going to be starting my own thread on the HDD monitoring thing, but just in case problem are curious. This is what I was trying to fix.

image

sometimes devices try to show overview graphs for data they don’t have. Does this device have processor data?

Yep the device has processor data. I just recently rebuilt the server but historic data should still be present as I haven’t touched the rrd folder at all.

I have no solution for this. But everything has returned back to normal. I was originally about to state that the migration and the downtime period was greater than the amount of time the graph was generating for the period. However, this was present in the 1 week, 1 month, 1 year, 2 year plots so I don’t know.

Maybe the mystical fairy came and helped me out one evening. Regardless, it’s working now. Attached below is a 24 hour plot that shows the 4 hour downtime we experienced when fixing this issue.

Cheers. Thanks for your help.