Replace RRDtool with another time-series database

Projects
1

Is there any interest to discuss? Is this the best place to do so? (and of course let me know if this is already an active discussion somewhere that I missed :laughing:)

I have seen some mention of this idea for awhile, mostly about:

  1. polling devices at different intervals (1 minute and 5 minute polling on the same server, Feature request: Custom polling frequency, Run difference poller groups:)
  2. updated graphs (Better rrd graphs, User survey results May 2017, https://github.com/librenms/librenms/projects/8#card-35736121)

I started a pull request (a hack at best) to expose RRD data via web API, but as @murrant suggested, it should be built as a proper class: https://github.com/librenms/librenms/pull/12086/. (I ended up building it as a plugin to librenms that converts rrdgraph queries to rrdxport queries and returns data in JSONTIME format, but again it’s “just okay”)

I see some projects using RRD as backend that have a better frontend graphing replacement - https://github.com/mbolli/nfsen-ng - that may be an easy way to switch the frontend away from RRD graphs, which would allow the time-series database to be replaced eventually.

There are some decent options for graphing libraries like dygraphs, chartjs, visjs.

Once the graphs are replaced, it would make sense to replace RRD with an existing supported TSDB - but which would be best as a primary replacement?

Another option could be to implement a setting in the LibreNMS web UI to use an existing Grafana installation as the graphing engine, which could use any supported time-series database. Embed/iframe? I’m not very familiar with Grafana, but I assume this is possible.

I would love to help with this any way I can, but I’m not sure I have the PHP experience to create a proper class definition - LibreNMS\Data\Store is a generic driver to store data, with specific implementations for each backend - we would need something like LibreNMS\Data\Retrieve as a generic driver to query data with a specific implementation for one (or more) other TSDB.

If there is interest, maybe we can plot out milestones/issues in Github for the steps needed to accomplish this. Looking forward to further discussion!

2 Likes
2

I’m on discord at hseitz#1414

3

Would love feedback on this. Trying to evaluate different JS graphing libraries is hard.

4

I’ve been shipping data to InfluxDB for a while. We keep short-term in RRD and view in LibreNMS, and then we go to Grafana for anything older than a few weeks.
It’s pretty simple to build a Grafana dashboard with dynamic variables that make selecting a particular graph very easy:

image
image1321×690 73.8 KB

It would be really cool to see a two-way integration between LibreNMS and InfluxDB so that InfluxDB can replace RRD completely, but that is probably a tall order. I am pretty happy with the arrangement we have in place now.

5

It is indeed a large project. I could use some help. I have a good start in the branch I linked above with basic graphs working in different JS libraries. It would likely break the current external TSDB support because it would need restructuring.

I was primarily working with RRD and InfluxDB in the PR, but my OS has updated to InfluxDB 2 and I’m considering reworking to use that. A little unsure there.

6

This is a great start! I like how your approach generalizes the queries between each backend with QueryBuilder. How do you translate the current rrd graph queries to rrd xport queries? or will each one have to be re-written?

7

I plan to change the rrd structure. Keeping one DS per rrd file will allow LibreNMS to be much more flexible with data storage. This means every graph will need to be re-written.

Should be easy enough to migrate one set of data at a time.

8

do you think RRD migration is possible via the update/daily scripts, or would it be a manual process for the end user?

9

Migration would be done by the poller, however, it will increase disk space usage as we will not remove the old rrds automatically. (the rrd cleanup option can be enabled to do that though)

10

This is brilliant and would open up so many possibilities, any way I can help to get the ball rolling?