I have a bunch of services monitored, inclding an AWS-monitoring that displays the estimated cost and total size of s3 buckets.
This works fine, and i have alerts and graphs all working nicely.
Yesterday I added a new bucket to AWS and that caused the graphs to not work anymore.
The issue turned out to be that the check_aws-script started returning another entry, so instead of returning
OK | cost=123.45 bucket_1=123456B bucket_2=54321B
It started returning
OK | cost=123.45 bucket_1=123456B bucket_2=54321B bucket_new=9876B
The rrd update command then started failing, because the RRD-file did not have a DS for bucket_new (fair enough).
So I then had to manually update the RRD-file with rrd tune to add the new DS.
After that, the old graphs started updating again, but I did not get a new graph for the new bucket.
To get that, I had to manually modify the database and update services set service_ds = ... "bucket_new":"GB"} where service_id=34
THEN I also got the new graphs in the WebGUI.
After looking at the code, I believe the last step could have been avoided by just changing the service_ds to either ‘{}’ or ‘’, and it should have been recreated automatically on the next service poll(?)
But there should really be a better way to do this. There should be some kind of detection that the response from the check_service has been modified, and does no longer match the RRD-definition, and a rrd tune should be run to automatically update the rrd-file. And then the correspoding service_ds field in the database should be cleared (so it will be auto-updated) or updated directly.
Also - while we are at it - I would love to be able to modify the displayed name in the graphs. It currently just takes the DS-name and do some safe-string magic and uppercases the first letter. And due to RRD-limitations the name can not be longer than 19 characters.
This works fine for lots of services, but for others, it would be very nice to be able to modify what is displayed as the DS is not always very human readable…