Nagios Plugin - check_dcm works from command line but not in LibreNMS

Hey there,

I want to monitor reachability of DICOM-Servers, which can be done via a nagios plugin. I was able to run it from the command line:

./check_dcm.py -H 192.168.11.11 -p 104
DICOM OK - Association Accepted
echo $?
0

So this should work. Equally, if I change the port to something that does not exist, it fails like expected:

./check_dcm.py -H 192.168.11.11 -p 11114
DICOM CRITICAL - Association Request Failed
echo $?
2

Within the LibreNMS Webinterface I was able to pick dcm.py as Service, added 192.168.11.11 as host and -p 104 as argument. This makes the Service go yellow and not display any output…

Can someone help? Thank you so much in advance :slight_smile:

This is the check_dcm.py I am using:

#!/usr/bin/python
# 
# check_dcm.py
# Nagios/Icinga plugin to check DICOM services.
# 
# The script is a wrapper for dcmtk's (http://dicom.offis.de/dcmtk.php.en) echoscu
# to monitor STORE SCP.  It will have to be installed (or the binary built) on the
# nagios system. Easiest way on an ubuntu system is to use apt:
# 
# sudo apt-get install dcmtk
# 
# Usage:
# You can hard code your port/ae_title but I prefer to use object variables as below
# (http://nagios.sourceforge.net/docs/3_0/customobjectvars.html)
# 
# define command {
#         command_name    check_dcm
#         command_line    $USER1$/check_dcm -H $HOSTADDRESS$ -p $_HOSTPORT$ -a $_HOSTAE_TITLE$ -v
#         }


import subprocess
import sys
import argparse

# Version
VERSION = '0.2.0'

# Exit Codes
STATE_OK = 0
STATE_WARNING = 1
STATE_CRITICAL = 2
STATE_UNKNOWN = 3

# Arguments
parser = argparse.ArgumentParser()
parser.add_argument("-V", "--version", action="store_true", help="display plugin version")
parser.add_argument("-v", "--verbosity", action="count", default=0, help="increase output verbosity")
parser.add_argument("-t", "--timeout", default='10', help="seconds before request timeout")
parser.add_argument("-aet", "--aetitle", help="calling AE Title (default: ECHOSCU)")
parser.add_argument("-aec", "--call", help="ae title of modality (default: ANY-SCP)")
parser.add_argument("-H", "--hostname", help="hostname of modality")
parser.add_argument("-p", "--port", default='104', help="tcp/ip port number of modality")


try:
	args = parser.parse_args()
except SystemExit(e):
	# bad args, so service state unknown
	sys.exit(STATE_UNKNOWN)
	raise e

if args.version:
	# service state unknown, but here's your version number.
	print(VERSION)
	sys.exit(STATE_UNKNOWN)

# build command line arguments
cmd = ['/usr/bin/echoscu']

if args.verbosity >= 3:
	cmd.append('--debug')
elif args.verbosity == 2:
	cmd.append('--verbose')

if args.timeout:
	cmd.append('-to') 
	cmd.append(args.timeout)

if args.aetitle:
	cmd.append('-aet')
	cmd.append(args.aetitle)

if args.call:
	cmd.append('-aec')
	cmd.append(args.call)

cmd.append(args.hostname)
cmd.append(args.port)


# send out a dicom ping and see what comes back
try:
	p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
except Exception(e):
	sys.exit(STATE_UNKNOWN)
else:
	stdout = p.communicate()[0]

	if p.returncode:
		if args.verbosity:
			print("DICOM CRITICAL - Association Request Failed (TCP Initialization Error: Connection refused)")
			if args.verbosity > 1:
				for line in stdout.splitlines():
					print(line)
		else:
			print("DICOM CRITICAL - Association Request Failed")

		sys.exit(STATE_CRITICAL)

	else:
		if args.verbosity:
			print("DICOM OK - Association Accepted (Received Echo Response (Status: Success))")
			if args.verbosity > 1:
				for line in stdout.splitlines():
					print(line)
		else:
			print("DICOM OK - Association Accepted")

		sys.exit(STATE_OK)

Things I would check are:

  • Did you manually test as “librenms”-user? The checks are executed with this account, so it could be limitations that user has.
  • Check if your script does not need any environment like path-resolution or exports, because that also could be missing when run from within another program.
  • If all does not help, simply add a print() statement that displays the whole commandline as run by LibreNMS. Maybe some parameter gets quirked when calling?

Hi @ChrisK928, thanks for your reply!

  1. Before running ./check_dcm.py [...] I changed to the librenms-user with su - librenms. Confirmed the change with whoami. That should be covered.

  2. How would I find that out? The only path I could find was /usr/bin/echoscu as this script is only a wrapper for the dicomtoolkit.

  1. Good idea. I would rather find the bug, but this might be an option to see what actually gets returned by the script when librenms tries to execute it.

For 2.: That is indeed hard to track if the script is calling another script that then maybe requires path-resolving or any environment-variables to be set.
You could create a dummy-check that just dumps the Linux “export” command to some file and compare that with what export shows when manually run from shell. If that doesn’t differ, it’s no profile- or environment-problem.

Okay, seems like I was able to find the problem. This check depends on echoscu being installed, which it is in /usr/bin/echoscu. Seems like LibreNMS is not able to reach this directory. Is that possible?

Here is what I got after modifying the script to be somewhat more verbose:

DICOM UNKNOWN - [Errno 2] No such file or directory: '/usr/bin/echoscu'

The file is definitely there and it is found when run from the command line as librenms. I am really confused.

@ChrisK928 do you know of any restrictions on which software can be executed from within librenms?

Hi @luc-ass
Seems that you need to find the correct combination of PATH, chmod and other rights, etc. This is a little bit annoying to troubleshoot (and frustrating).
Nothing directly linked to LibreNMS here, only Linux stuff.

I also second that it must be some problems with invocation of the external command. I would try if using any alternative to subprocess.popen works better.
For example:

p = subprocess.check_output([cmd], shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

Maybe adding “shell=True” is also sufficient and keep using popen, but I’m not sure.

HTH!

I really appreciate all your input!

By adding shell=True I was able to get a little further. The output still differs between (1) cli and (2) Librenms:

  1. DICOM OK - Association Accepted
  2. DICOM CRITICAL - Association Request Failed

So I added print(os.environ) to see what might be different:

CLI
{
   "USER":"librenms",
   "SHLVL":"1",
   "HOME":"/home/librenms",
   "OLDPWD":"/home/librenms",
   "PAGER":"less",
   "LOGNAME":"librenms",
   "TERM":"xterm",
   "LC_COLLATE":"C",
   "PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
   "LANG":"C.UTF-8",
   "SHELL":"/bin/sh",
   "PWD":"/data/monitoring-plugins",
   "CHARSET":"UTF-8"
}
LibreNMS:
{
   "REDIS_PORT":"6379",
   "HOSTNAME":"librenms-dispatcher",
   "APP_URL":"/",
   "DB_PORT":"3306",
   "REDIS_DB":"0",
   "SHLVL":"1",
   "HOME":"/root",
   "CWD":"/opt/librenms",
   "DB_NAME":"librenms",
   "DB_DATABASE":"librenms",
   "PGID":"1000",
   "UPLOAD_MAX_SIZE":"16M",
   "LIBRENMS_WEATHERMAP":"false",
   "SESSION_DRIVER":"redis",
   "DB_USERNAME":"librenms",
   "LIBRENMS_DOCKER":"1",
   "CACHE_DRIVER":"redis",
   "PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
   "SIDECAR_DISPATCHER":"1",
   "S6_BEHAVIOUR_IF_STAGE2_FAILS":"2",
   "NODE_ID":"dispatcher1",
   "PUID":"1000",
   "MAX_INPUT_VARS":"1000",
   "LOG_IP_VAR":"remote_addr",
   "REDIS_PASSWORD":"",
   "REAL_IP_HEADER":"X-Forwarded-For",
   "REAL_IP_FROM":"0.0.0.0/32",
   "LIBRENMS_PATH":"/opt/librenms",
   "REDIS_HOST":"redis",
   "DB_PASSWORD":"***",
   "APP_KEY":"***",
   "PWD":"/opt/librenms",
   "LIBRENMS_SNMP_COMMUNITY":"librenmsdocker",
   "OPCACHE_MEM_SIZE":"128",
   "LIBRENMS_WEATHERMAP_SCHEDULE":"*/5 * * * *",
   "DB_TIMEOUT":"60",
   "REDIS_SCHEME":"tcp",
   "LC_NUMERIC":"C",
   "TZ":"Europe/Berlin",
   "DB_HOST":"db",
   "DB_USER":"librenms",
   "MEMORY_LIMIT":"256M",
   "DISPATCHER_NODE_ID":"dispatcher1"
}

Noably user is missing from the LibreNMS version.

This looks like the command is executed using “sudo”. Maybe that’s the problem here?

Other idea would be to check if SELinux / Apparmor could have blocked the invocation of that file. If you have one of them running, you could try to temporarly disable the service of switch to “complain” mode, so that the potential violation will only get logged instead of being blocked:

https://linuxconfig.org/how-to-disable-apparmor-on-ubuntu-20-04-focal-fossa-linux