Rrdtool zombies when using API to export graphs

Hi guys.

I am using the API to export graphs on a web page hosted on a remote server. I do this through cron. Every time the cron file is executed on the remote server I get two zombies of rrdtool on the libreNMS server. The remote web server executes the following :

curl -H ‘X-Auth-Token: myAIPkey’ ‘http://librenms.server/api/v0/devices/10.70.25.30/device_processor?width=500&height=200’ > /var/www/html/exports/system1_cpu.png
curl -H ‘X-Auth-Token: myAIPkey’ ‘http://librenms.server/api/v0/devices/10.70.25.31/device_processor?width=500&height=200’ > /var/www/html/exports/system2_cpu.png

Any ideas ?

nginx : nginx/1.10.2
php : PHP 7.0.14 (fpm-fcgi)

what version of librenms?

Ver 1.23
It’s on ab03af8

confirmed…

Can you test this patch please:

diff --git a/html/includes/api_functions.inc.php b/html/includes/api_functions.inc.php
index 44ddb58..3d1723b 100644
--- a/html/includes/api_functions.inc.php
+++ b/html/includes/api_functions.inc.php
@@ -73,7 +73,9 @@ function get_graph_by_port_hostname()
     $auth           = '1';
     $vars['id']     = dbFetchCell("SELECT `P`.`port_id` FROM `ports` AS `P` JOIN `devices` AS `D` ON `P`.`device_id` = `D`.`device_id` WHERE `D`.`hostname`=? AND `P`.`$port`=?", array($hostname, $vars['port']));
     $app->response->headers->set('Content-Type', 'image/png');
+    rrdtool_initialize(false);
     include 'includes/graphs/graph.inc.php';
+    rrdtool_close();
 }


@@ -148,7 +150,10 @@ function get_graph_generic_by_hostname()
     $auth           = '1';
     $vars['device'] = dbFetchCell('SELECT `D`.`device_id` FROM `devices` AS `D` WHERE `D`.`hostname`=?', array($hostname));
     $app->response->headers->set('Content-Type', 'image/png');
+    rrdtool_initialize(false);
     include 'includes/graphs/graph.inc.php';
+    rrdtool_close();
+
 }


@@ -502,7 +507,9 @@ function get_graph_by_portgroup()
     $vars['type'] = 'multiport_bits_separate';
     $vars['id']   = $if_list;
     $app->response->headers->set('Content-Type', 'image/png');
+    rrdtool_initialize(false);
     include 'includes/graphs/graph.inc.php';
+    rrdtool_close();
 }


I’ve installed the patch. From a quick look, it seems that solves the issue. I will leave it for 30 minutes ( I generate graphs every 5 minutes via cron ) and I ll get back

1 Like

Ok something strange is happening…

I executed the graph script from the remote server manually multiple times and no zombie process appeared that time. After a while zombies appeared again.

nginx 9801 8554 0 19:56 ? 00:00:00 [rrdtool]
nginx 12157 8556 0 20:06 ? 00:00:00 [rrdtool]
nginx 12170 8605 0 20:06 ? 00:00:00 [rrdtool]
nginx 13787 8623 0 20:12 ? 00:00:00 [rrdtool]
nginx 13788 8605 0 20:12 ? 00:00:00 [rrdtool]
nginx 13789 12153 0 20:12 ? 00:00:00 [rrdtool]
nginx 13790 8557 0 20:12 ? 00:00:00 [rrdtool]
nginx 13791 12147 0 20:12 ? 00:00:00 [rrdtool]

According to the remote server’s cron, the script run at :

Jan 25 20:05:01 hera CROND[20732]: (root) CMD (/Scripts/graphs.sh &> /dev/null 2>&1)
Jan 25 20:10:01 hera CROND[20775]: (root) CMD (/Scripts/graphs.sh &> /dev/null 2>&1)
Jan 25 20:15:01 hera CROND[20824]: (root) CMD (/Scripts/graphs.sh &> /dev/null 2>&1)

So these zombies are not from the script.
The librenms server’s cron says :

Jan 25 20:06:01 ares CROND[12126]: (librenms) CMD ( /opt/librenms/alerts.php >> /dev/null 2>&1)
Jan 25 20:07:01 ares CROND[12185]: (librenms) CMD ( /opt/librenms/alerts.php >> /dev/null 2>&1)
Jan 25 20:08:01 ares CROND[12328]: (librenms) CMD ( /opt/librenms/alerts.php >> /dev/null 2>&1)
Jan 25 20:09:01 ares CROND[12463]: (librenms) CMD ( /opt/librenms/alerts.php >> /dev/null 2>&1)
Jan 25 20:10:01 ares CROND[12562]: (librenms) CMD ( /opt/librenms/alerts.php >> /dev/null 2>&1)
Jan 25 20:10:01 ares CROND[12563]: (librenms) CMD ( /opt/librenms/check-services.php >> /dev/null 2>&1)
Jan 25 20:10:01 ares CROND[12566]: (librenms) CMD ( /opt/librenms/discovery.php -h new >> /dev/null 2>&1)
Jan 25 20:10:01 ares CROND[12567]: (librenms) CMD ( /opt/librenms/poll-billing.php >> /dev/null 2>&1)
Jan 25 20:10:01 ares CROND[12569]: (librenms) CMD ( /opt/librenms/cronic /opt/librenms/poller-wrapper.py 16)
Jan 25 20:11:01 ares CROND[13685]: (librenms) CMD ( /opt/librenms/alerts.php >> /dev/null 2>&1)
Jan 25 20:12:01 ares CROND[13745]: (librenms) CMD ( /opt/librenms/alerts.php >> /dev/null 2>&1)

So the graph script does not create zombies anymore but something else does. I will disable the script and I will try to find what causes this

Strange…

nginx 19630 18856 0 20:35 ? 00:00:00 [rrdtool]
nginx 20055 18857 0 20:36 ? 00:00:00 [rrdtool]
nginx 20126 19540 0 20:36 ? 00:00:00 [rrdtool]
nginx 20199 19718 0 20:37 ? 00:00:00 [rrdtool]
nginx 20218 19500 0 20:37 ? 00:00:00 [rrdtool]
nginx 20280 18856 1 20:37 ? 00:00:00 [rrdtool]

A zombie is created everytime I click a device from the device list. If I go back and click the same device again, no zombie will be created. If I go back on the list and click another device a zombie will appear

So the issue of zombies and API is resolved by the patch posted bu @laf . But another one exists

I can’t make it happen for me so unless you can narrow it down a bit more then I’m not sure I can suggest anything else.

Last time I looked and fixed zombie processes I found a repeatable way with graphs that errored, if you find some way to repeat this then report back. The above process you said above doesn’t work for me.

It’s also worth mentioning version numbers for everything including rrdtool

I will try to gather as much as possible hopefully this weekend and I will post back my findings.