Graphs stopped working after adding more hosts

[Whiskey-Water]

Hello - My Libre was working perfectly since initial setup about 30 days or so. Today I added about 10 more hosts to bring it to 40 and some time while adding those hosts it just stopped graphing and I have no idea why. My validate config looks great! When I look at the logs it appears to be polling devices like crazy which I think is good however there is no graph data for some unknown reason. I rebooted the server but no luck. Again the only change is adding more devices.

====================================

Component	Version
LibreNMS	22.1.0-19-ga1c6b735b
DB Schema	2021_11_29_165436_improve_ports_search_index (229)
PHP	7.4.3
Python	3.8.10
MySQL	10.3.32-MariaDB-0ubuntu0.20.04.1
RRDTool	1.7.2
SNMP	5.8
====================================

[OK] Composer Version: 2.2.5
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct

If it’s an issue with the WebUI then please consider including a screenshot and the browser version you are using.

[Whiskey-Water]

[Whiskey-Water]

[laf]

You need to post the output of ./poller.php -h HOSTNAME -d -v

[Whiskey-Water]

@laf

./poller.php -h HOSTNAME -d -v

https://p.libren.ms/view/1f3da251

[laf]

Looks to be recording the data ok (I’m assuming you aren’t running out of disk space?)

If you click one of the broken graphs, then click show RRD Command and copy / paste the output here.

[Whiskey-Water]

RRDTool Command

rrdtool graph /tmp/ZH2UuE1cc8oefihI -g --alt-autoscale-max --rigid -E --start 1643238900 --end 1643325300 --width 1728 --height 484 -c BACK#EEEEEE00 -c SHADEA#EEEEEE00 -c SHADEB#EEEEEE00 -c CANVAS#FFFFFF00 -c GRID#a5a5a5 -c MGRID#FF9999 -c FRAME#5e5e5e -c ARROW#5e5e5e -R normal -c FONT#000000 --font LEGEND:8:DejaVuSansMono --font AXIS:7:DejaVuSansMono --font-render-mode normal COMMENT:’ ’ COMMENT:’ Current’ COMMENT:’ Average’ COMMENT:’ Maximum’ COMMENT:’\n’ DEF:inB0=127.0.0.1/port-id76.rrd:INOCTETS:AVERAGE DEF:outB0=127.0.0.1/port-id76.rrd:OUTOCTETS:AVERAGE CDEF:octets0=inB0,outB0,+ CDEF:inbits0=inB0,8,* CDEF:outbits0=outB0,8,* CDEF:outbits0_neg=outbits0,-1,* CDEF:bits0=inbits0,outbits0,+ AREA:inbits0#CAE853:‘ens32 In’ GPRINT:inbits0:LAST:%6.2lf%sbps GPRINT:inbits0:AVERAGE:%6.2lf%sbps GPRINT:inbits0:MAX:%6.2lf%sbps COMMENT:’\n’ HRULE:999999999999999#CC7CCC:’ Out’ GPRINT:outbits0:LAST:%6.2lf%sbps GPRINT:outbits0:AVERAGE:%6.2lf%sbps GPRINT:outbits0:MAX:%6.2lf%sbps COMMENT:’\n’ AREA:outbits0_neg#CC7CCC: HRULE:0#999999 --daemon unix:/var/run/rrdcached.sock

[Whiskey-Water]

[Whiskey-Water]

I must have a real stumper on here? @laf that is just my luck! Thanks

[Kevin_Krumm]

Check your librenms.logs

[Whiskey-Water]

I have a ton of “devices polled” in the /opt/librenms/logs/librenms.log. Which are current and seem totally normal.

[Kevin_Krumm]

Are using RRDCache?

[Whiskey-Water]

I “think so” based upon it being a pretty default install. Is there a good way to know for sure?

[Whiskey-Water]

I did figure out the problem. My workers seconds were way out of balance. I added more CPU and then increased the “Poller Workers” under Poller settings in the GUi to resolve the issue.
I am curious however when you mention RRDCache. Is there a way to check if I am using that and if it is healthy? Thank you

[Kevin_Krumm]

Yes check out RRDCached - LibreNMS Docs