I opened an issue on github for rrdtool also https://github.com/oetiker/rrdtool-1.x/issues/1277
I only post this to here because maybe someone have some experience about this scale.
I have two LibreNMS instance, each produces and updates 500-800k rrd files. When I restart any of them, rrdcached starts very slow, it takes ~2 hours.
command line options:
/usr/bin/rrdcached -B -F -R -w 1800 -z 1800 -l 0:42217 -b /opt/librenms/rrd/ -j /var/lib/rrdcached/journal/ -G librenms -U librenms -p /run/rrdcached.pid -l unix:/run/rrdcached.sock
RRD files is on 1TByte XFS volume, blkid:
/dev/vdb: LABEL="RRD_XFS" UUID="78752a7b-bd48-4aa3-bbc2-01928c1695e8" BLOCK_SIZE="4096" TYPE="xfs"
When rrdcached starting up, I see 50-70 reading IOPs with 300-400kByte/sec I/O BW. After rrdcached started everything works fine.
First of all I need to find out what causes this very slow startup. Any opinion would be appreciated. Maybe I need to turn off the journal in rrdcached? It would be help if I move from XFS to ZFS and using persistent L2ARC?
Thanks!
Best Regards,
Tibor