Broken Graphs

[santoshkotla]

Hi,

I am deployed Librenms with 25 Cores, 50GB RAM and 500GB of space.

We have completed migrating devices into this instance and I saw that the CPU was hitting 90%. I deployed another poller just with Librenms without the NGINX or the Database and wanted it to be as a poller to the main instance. Ever since that deployment the graphs are broken. I thought it may have been because a misconfig on the poller and I commented out the RRDCACHED config statements and removed the necessary commands from the poller .env file.

Even after this the main instance still has broken graphs.

Validate and the other processes seem to work just fine. Any help in this regard would be much appreciated.

Thanks,
Santosh Kotla

[santoshkotla]

librenms@dc5lp-vlibrenms01:~$ ./validate.php

Component	Version
LibreNMS	22.1.0-19-ga1c6b735b
DB Schema	2021_25_01_0129_isis_adjacencies_nullable (229)
PHP	7.4.3
Python	3.8.10
MySQL	10.5.13-MariaDB-1:10.5.13+maria~focal
RRDTool	1.7.2
SNMP	5.8

====================================

[OK] Composer Version: 2.2.5
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct

###########################################################

root@dc5lp-vlibrenms01:~# systemctl status rrdcached
● rrdcached.service - LSB: start or stop rrdcached
Loaded: loaded (/etc/init.d/rrdcached; generated)
Active: active (running) since Thu 2022-01-27 15:47:27 UTC; 3h 17min ago
Docs: man:systemd-sysv-generator(8)
Process: 1857526 ExecStart=/etc/init.d/rrdcached start (code=exited, status=0/SUCCESS)
Tasks: 441 (limit: 60139)
Memory: 2.5G
CGroup: /system.slice/rrdcached.service
└─1857616 /usr/bin/rrdcached -l 0:42217 -R -j /var/lib/rrdcached/journal/ -F -b /opt/librenms/rrd -B -w 1800 -z 900 -w 1800 -z 1800 -t 4 -b /opt/librenms/rrd/ -j /var/lib/rrdca>

Jan 27 15:47:27 dc5lp-vlibrenms01 rrdcached[1857526]: * rrdcached started
Jan 27 15:47:27 dc5lp-vlibrenms01 systemd[1]: Started LSB: start or stop rrdcached.
Jan 27 15:59:25 dc5lp-vlibrenms01 rrdcached[1857616]: handle_request_update: Could not read RRD file.
Jan 27 16:04:02 dc5lp-vlibrenms01 rrdcached[1857616]: handle_request_update: Could not read RRD file.
Jan 27 17:23:39 dc5lp-vlibrenms01 rrdcached[1857616]: handle_request_update: Could not read RRD file.
Jan 27 17:58:05 dc5lp-vlibrenms01 rrdcached[1857616]: handle_request_update: Could not read RRD file.
Jan 27 18:13:46 dc5lp-vlibrenms01 rrdcached[1857616]: handle_request_update: Could not read RRD file.
Jan 27 18:23:41 dc5lp-vlibrenms01 rrdcached[1857616]: handle_request_update: Could not read RRD file.
Jan 27 18:28:38 dc5lp-vlibrenms01 rrdcached[1857616]: handle_request_update: Could not read RRD file.
Jan 27 18:57:34 dc5lp-vlibrenms01 rrdcached[1857616]: handle_request_update: Could not read RRD file.

[laf]

Have you followed the instructions for running distributed polling? You can’t disable rrdcached in that type of setup and their are some pre-requisites to follow.

[santoshkotla]

Hi @laf ,

Thanks for the response.

Yes. I followed the documentation when I was setting up the main poller and everything was just fine. No issues at all. We had about 1500 devices on that single instance and it was putting up just fine.

Until I setup a poller and configured the .env file. Things started to break and I took off the poller. Now I am trying to get the main instance to work as it was earlier.

Thanks,
Santosh Kotla

[santoshkotla]

There were three spaces on the line before php in the start of my config file and that caused the graphs to break.

This issue is now resolved.

Thanks,
Santosh Kotla