Docker stats application not drawing graphs

hugalafutro · 1 April 2021 22:25

Hi,
Trying to get https://docs.librenms.org/Extensions/Applications/#docker-stats working. On the client machine(s) I confirmed the script works by manually running it and it outputs json with the docker data. Librenms discovers it as new app, says status ok, but all the graphs say Error drawing graph.

Example of one such graph’s RRD command:

RRDTool Command

rrdtool graph /tmp/EQjsqQ9LbNDGsf7E  --alt-autoscale-max --rigid -E --start 1617228900 --end 1617315581 --width 2273 --height 648 -c BACK#EEEEEE00 -c SHADEA#EEEEEE00 -c SHADEB#EEEEEE00 -c CANVAS#FFFFFF00 -c GRID#a5a5a5 -c MGRID#FF9999 -c FRAME#5e5e5e -c ARROW#5e5e5e -R normal -c FONT#000000 --font LEGEND:8:DejaVuSansMono --font AXIS:7:DejaVuSansMono --font-render-mode normal COMMENT:'PIDs (count)                       Now         Min         Max        Avg\l' COMMENT:'\l'
RRDTool Output

OK u:0.00 s:0.00 r:0.00

output of the script on one of clients:

[email protected]:~ $ /etc/snmp/docker-stats.sh
{"version":"1","data":[{"container":"zigbee2mqtt","pids":22,"memory":{"used":"0B","limit":"0B","perc":"0.00%"},"cpu":"0.00%"},{"container":"mosquitto","pids":1,"memory":{"used":"0B","limit":"0B","perc":"0.00%"},"cpu":"0.10%"},{"container":"ddns-updater","pids":11,"memory":{"used":"0B","limit":"0B","perc":"0.00%"},"cpu":"0.00%"},{"container":"room-assistant","pids":10,"memory":{"used":"0B","limit":"0B","perc":"0.00%"},"cpu":"9.36%"},{"container":"diun","pids":16,"memory":{"used":"0B","limit":"0B","perc":"0.00%"},"cpu":"0.03%"},{"container":"pihole","pids":21,"memory":{"used":"0B","limit":"0B","perc":"0.00%"},"cpu":"0.15%"},{"container":"portainer_agent","pids":10,"memory":{"used":"0B","limit":"0B","perc":"0.00%"},"cpu":"0.00%"},{"container":"portainer","pids":13,"memory":{"used":"0B","limit":"0B","perc":"0.00%"},"cpu":"0.00%"},{"container":"zigbee2mqtt-assistant","pids":21,"memory":{"used":"0B","limit":"0B","perc":"0.00%"},"cpu":"1.88%"}],"error":"0","errorString":""}

Running latest LibreNMS inside a docker. All other Applications I use work ok (fail2ban, freebsdnfsserver, ntp-server, osupdate, smart, unbound, ups-apcupsd)

Any advice welcome, thanks!

Jellyfrog · 2 April 2021 00:08

I’m guessing you have this problem; https://docs.librenms.org/Extensions/Applications/#sudo

Check what the discovery/poller says with debug enabled.

hugalafutro · 2 April 2021 00:32

Having dealt with this before for raspberry.sh I thought it would be that but it ain’t, however it pointed me the right way (sorta).

Executing sudo -u Debian-snmp /etc/snmp/docker-stats.sh yields

{"version":"1","data":[],"error":"1","errorString":"Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version: dial unix /var/run/docker.sock: connect: permission denied"}

Adding the script to sudoers as such Debian-snmp ALL = NOPASSWD: /etc/snmp/raspberry.sh, /usr/bin/vcgencmd, /etc/snmp/docker-stats.sh however does not solve it.

What solves it is running sudo chmod 662 /var/run/docker.sock which makes the docker socket writable to everyone.

This sounds like it could have security implications though, nor do I know if it will survive restart. Is there different way to achieve same result?

Jellyfrog · 2 April 2021 00:40

Did you add sudo to the extend line?

hugalafutro · 2 April 2021 00:48

You’re right I didn’t. Restoring the docker.sock to 660, adding the script to sudoers and modifying the extend to include sudo solves it.

Thanks!

willhseitz · 8 April 2021 16:58

I’m having a similar issue where Docker Stats and Certificate apps are showing “Error Drawing Graph”. Other application graphs are working. RRDtool command shows similar as OP and missing the RRD files:

RRDTool Command

rrdtool graph /tmp/S0kW1Ye7fWc2B0ZB  -l 0 -E --start 1617813900 --end 1617900572 --width 1692 --height 517 -c BACK#EEEEEE00 -c SHADEA#EEEEEE00 -c SHADEB#EEEEEE00 -c CANVAS#FFFFFF00 -c GRID#292929 -c MGRID#2f343e -c FRAME#5e5e5e -c ARROW#5e5e5e -R normal -c FONT#7f7f7f --font LEGEND:8:DejaVuSansMono --font AXIS:7:DejaVuSansMono --font-render-mode normal COMMENT:'Age (days)                         Now         Min         Max        Avg\l' COMMENT:'\l' --daemon librenms-rrd.myorgdomain.net:42217

RRDTool Output

OK u:0.01 s:0.01 r:0.00

I fixed the permissions issue of Debian-snmp running the docker-stats.sh script, and a debug polling run shows data is being returned correctly:

: ..
SNMP['/usr/bin/snmpget' '-v2c' '-c' 'COMMUNITY' '-Oqv' '-m' 'NET-SNMP-EXTEND-MIB' '-M' '/opt/librenms/mibs' 'udp:HOSTNAME:161' 'nsExtendOutputFull.6.100.111.99.107.101.114']
{"version":"1","data":[{"container":"netbox-docker_nginx_1","pids":2,"memory":{"used":"2.078MiB","limit":"1.941GiB","perc":"0.10%"},"cpu":"0.00%"},{"container":"netbox-docker_netbox_1","pids":4,"memory":{"used":"52.35MiB","limit":"1.941GiB","perc":"2.63%"},"cpu":"0.10%"},{"container":"net
box-docker_netbox-worker_1","pids":2,"memory":{"used":"12.24MiB","limit":"1.941GiB","perc":"0.62%"},"cpu":"0.10%"},{"container":"netbox-docker_redis_1","pids":6,"memory":{"used":"2.699MiB","limit":"1.941GiB","perc":"0.14%"},"cpu":"0.22%"},{"container":"netbox-docker_postgres_1","pids":8,"
memory":{"used":"7.758MiB","limit":"1.941GiB","perc":"0.39%"},"cpu":"1.34%"},{"container":"netbox-docker_redis-cache_1","pids":5,"memory":{"used":"1.535MiB","limit":"1.941GiB","perc":"0.08%"},"cpu":"0.22%"}],"error":"0","errorString":""}

RRD[last 192.168.100.143/app-docker-348-netbox-docker_nginx_1.rrd  --daemon librenms-rrd.myorgdomain.net:42217]
RRD[update 192.168.100.143/app-docker-348-netbox-docker_nginx_1.rrd N:0:2:2084132880:2178940:0.1 --daemon librenms-rrd.myorgdomain.net:42217]
RRD[last 192.168.100.143/app-docker-348-netbox-docker_netbox_1.rrd  --daemon librenms-rrd.myorgdomain.net:42217]
RRD[update 192.168.100.143/app-docker-348-netbox-docker_netbox_1.rrd N:0.1:4:2084132880:54892953:2.63 --daemon librenms-rrd.myorgdomain.net:42217]
RRD[last 192.168.100.143/app-docker-348-netbox-docker_netbox-worker_1.rrd  --daemon librenms-rrd.myorgdomain.net:42217]                                                                                                                                                                            RRD[update 192.168.100.143/app-docker-348-netbox-docker_netbox-worker_1.rrd N:0.1:2:2084132880:12834570:0.62 --daemon librenms-rrd.myorgdomain.net:42217]
RRD[last 192.168.100.143/app-docker-348-netbox-docker_redis_1.rrd  --daemon librenms-rrd.myorgdomain.net:42217]
RRD[update 192.168.100.143/app-docker-348-netbox-docker_redis_1.rrd N:0.22:6:2084132880:2830106:0.14 --daemon librenms-rrd.myorgdomain.net:42217]
RRD[last 192.168.100.143/app-docker-348-netbox-docker_postgres_1.rrd  --daemon librenms-rrd.myorgdomain.net:42217]
RRD[update 192.168.100.143/app-docker-348-netbox-docker_postgres_1.rrd N:1.34:8:2084132880:8134852:0.39 --daemon librenms-rrd.myorgdomain.net:42217]
RRD[last 192.168.100.143/app-docker-348-netbox-docker_redis-cache_1.rrd  --daemon librenms-rrd.myorgdomain.net:42217]
RRD[update 192.168.100.143/app-docker-348-netbox-docker_redis-cache_1.rrd N:0.22:5:2084132880:1609564:0.08 --daemon librenms-rrd.myorgdomain.net:42217]
SQL[UPDATE `applications` set `app_state`=?,`app_status`=?,`timestamp`=NOW() WHERE `app_id` = ? ["OK","",348] 2.39ms]

even manually querying from RRDtool returns values for those RRD files:

$ sudo -u librenms rrdtool last 192.168.100.143/app-docker-348-netbox-docker_redis-cache_1.rrd  --daemon librenms-rrd.myorgdomain.net:42217
1617900840

LibreNMS web and pollers are all on 21.3.0. Any suggestions for where to look next?

hugalafutro · 11 April 2021 11:07

This is just as on the off-chance you’re running into this issue
https://github.com/librenms/docker/issues/124, but have you tried disabling rrdcached ?

Jellyfrog · 11 April 2021 16:29

Yes if you run rrdcached You might run into problems, currently being fixed in

github.com/librenms/librenms

Rrdtool array class

librenms:master ← djamp42:rrdtool-array-class

opened 12:44PM - 05 Apr 21 UTC

djamp42

+74 -2

Some applications grab an array of rrd filenames to display links on the webgui …and display the rrd graph data. This does not work over rrdcached, this pull request is to fix that.. I only have docker fixed in this pull request, but can fix the rest if this looks good. Here is the orginal PR that i tried. https://github.com/librenms/librenms/pull/12689 DO NOT DELETE THE UNDERLYING TEXT #### Please note > Please read this information carefully. You can run `./lnms dev:check` to check your code before submitting. - [X] Have you followed our [code guidelines?](https://docs.librenms.org/Developing/Code-Guidelines/) - [X] If my Pull Request does some changes/fixes/enhancements in the WebUI, I have inserted a screenshot of it. #### Testers If you would like to test this pull request then please run: `./scripts/github-apply <pr_id>`, i.e `./scripts/github-apply 5926` After you are done testing, you can remove the changes with `./scripts/github-remove`. If there are schema changes, you can ask on discord how to revert.

system · 18 April 2021 16:29

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.