All graphs stopped working

We have been using librenms without any issues since early June 2019. All graphs worked, as far as I can tell, up until 10/15. Now all graphs for all devices are red boxes. No changes to the system have been made in a while.

Here is the output for the RRD Command:

RRDTool Command

rrdtool graph /tmp/FUWoAcL4sTNKFigH --alt-autoscale-max --rigid -E --start 1571253600 --end 1571340000 --width 1547 --height 300 -c BACK#EEEEEE00 -c SHADEA#EEEEEE00 -c SHADEB#EEEEEE00 -c CANVAS#FFFFFF00 -c GRID#a5a5a5 -c MGRID#FF9999 -c FRAME#5e5e5e -c ARROW#5e5e5e -R normal -c FONT#000000 --font LEGEND:8:DejaVuSansMono --font AXIS:7:DejaVuSansMono --font-render-mode normal -u 100 -l 0 -E -b 1024 COMMENT:’ Min Cur Max\n’ DEF:mempoolfree1=XXXXXXXXX/mempool-cmp-1.rrd:free:AVERAGE DEF:mempoolused1=XXXXXXXXX/mempool-cmp-1.rrd:used:AVERAGE CDEF:mempooltotal1=mempoolused1,mempoolused1,mempoolfree1,+,/,100,* AREA:mempooltotal1#CC000010 DEF:mempoolfree2=XXXXXXXXX/mempool-cmp-7.rrd:free:AVERAGE DEF:mempoolused2=XXXXXXXXX/mempool-cmp-7.rrd:used:AVERAGE CDEF:mempooltotal2=mempoolused2,mempoolused2,mempoolfree2,+,/,100,* AREA:mempooltotal2#008C0010 LINE1:mempooltotal1#CC0000:'Processor ’ GPRINT:mempooltotal1:MIN:%3.0lf%% GPRINT:mempooltotal1:LAST:%3.0lf%% GPRINT:mempooltotal1:MAX:%3.0lf%%\l LINE1:mempooltotal2#008C00:'IOS Process stack ’ GPRINT:mempooltotal2:MIN:%3.0lf%% GPRINT:mempooltotal2:LAST:%3.0lf%% GPRINT:mempooltotal2:MAX:%3.0lf%%\l HRULE:0#999999 --daemon unix:/var/run/rrdcached.sock
RRDTool Output

ERROR: Unable to connect to rrdcached: Connection refused

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Here is the output for the ./validate.php

====================================

Component Version
LibreNMS 1.52-70-gf3ba894
DB Schema 2019_05_30_225937_device_groups_rewrite (135)
PHP 7.2.22
MySQL 5.5.64-MariaDB
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 1.9.0
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[WARN] Your install is over 24 hours out of date, last update: Sun, 23 Jun 2019 05:29:12 +0000
[FIX]:
Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.
[WARN] Your local git contains modified files, this could prevent automatic updates.
[FIX]:
You can fix this with ./scripts/github-remove
Modified Files:
bootstrap/cache/.gitignore
logs/.gitignore
rrd/.gitignore
storage/app/.gitignore
storage/app/public/.gitignore
storage/debugbar/.gitignore
storage/framework/cache/.gitignore
storage/framework/cache/data/.gitignore
storage/framework/sessions/.gitignore
storage/framework/testing/.gitignore
storage/framework/views/.gitignore
storage/logs/.gitignore
[FAIL] We have found some files that are owned by a different user than librenms, this will stop you updating automatically and / or rrd files being updated causing graphs to fail.
[FIX]:
sudo chown -R librenms:librenms /opt/librenms
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
Files:
/opt/librenms/composer.phar
/opt/librenms/html/plugins/Weathermap/.git/refs/heads/master
/opt/librenms/html/plugins/Weathermap/.git/refs/remotes/origin/master
/opt/librenms/html/plugins/Weathermap/.git/objects/4d
/opt/librenms/html/plugins/Weathermap/.git/objects/4d/b645d42f3df582bb2ea7580d4549147e8e5149
/opt/librenms/html/plugins/Weathermap/.git/objects/4d/c87ec34e177d12a88ddaeadc56607d6848ce30
/opt/librenms/html/plugins/Weathermap/.git/objects/9d
/opt/librenms/html/plugins/Weathermap/.git/objects/9d/fcbe2059c85980fa0623f08e0c4d899148cd7f
/opt/librenms/html/plugins/Weathermap/.git/objects/d0
/opt/librenms/html/plugins/Weathermap/.git/objects/d0/45731775a8c9f8fe5457dbbe8078f2731415f8
/opt/librenms/html/plugins/Weathermap/.git/objects/bd
/opt/librenms/html/plugins/Weathermap/.git/objects/bd/87d23165b351982cb6116792e5ae7d742a55a3
/opt/librenms/html/plugins/Weathermap/.git/objects/d2
/opt/librenms/html/plugins/Weathermap/.git/objects/d2/773150eeed7abdb09c2818bcba25723f82c247
/opt/librenms/html/plugins/Weathermap/.git/objects/b9
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

And here is the output of the daily.sh:

opt/librenms/daily.sh
Re-running /opt/librenms/daily.sh as librenms user
Updating SQL-Schema OK
Cleaning up DB OK

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I did not try to change ownership of the files listed in the validate.php output because this has always reported as a fail with no issues.

ERROR: Unable to connect to rrdcached: Connection refused

Your rrdcached service isn’t running/available.

Also your install is way out of date. You may need to run git pull && ./daily.sh

That was an easy fix, thanks.

systemctl restart rrdcached.service