I have noticed a strange issue when using the official LibreNMS docker image.
Recently, I have noticed that devices will just stop polling until I restart the dispatcher container. When I took a deeper look, I saw this error appear only once, just after the last successful poll.
A few recent posts with the exact issue, all unresolved.
Python version inside the docker container is 3.8.5.
dispatcher_1 | Poller_0-4(INFO):Completed poller run for 6 in 3.09s
dispatcher_1 | Poller_0-19(INFO):Completed poller run for 7 in 3.10s
dispatcher_1 | service.start()
dispatcher_1 | File "/opt/librenms/LibreNMS/service.py", line 404, in start
dispatcher_1 | sleep(self.config.master_resolution)
dispatcher_1 | File "/opt/librenms/LibreNMS/service.py", line 550, in reap
dispatcher_1 | handler = signal(SIGCHLD, SIG_DFL)
dispatcher_1 | File "/usr/lib/python3.8/signal.py", line 47, in signal
dispatcher_1 | handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
dispatcher_1 | TypeError: 'int' object is not callable
dispatcher_1 | Poller_0-23(INFO):Completed poller run for 1 in 2.68s
Wondering if you have resolved this issue somehow. I have just upgraded docker image and moved from cron to dispatcher and poller seems to stop polling (every few hours) after the same python stack trace. Restarting the container will cause polling to begin again, at least for some time.
[OK] Installed from the official Docker image; no Composer required
[OK] Database connection successful
[OK] Database schema correct
[WARN] IPv6 is disabled on your server, you will not be able to add IPv6 devices.
[WARN] Updates are managed through the official Docker image
It hasn’t happened for ages now. The only thing I changed was a dodgy alerting rule for average CPU across all cores. It was throwing SQL related errors in the logs somewhere. I took that out and it hasn’t happened again, though the gaps between it happening varied from days to weeks, so it could still happen again.
If you have any custom/SQL query alerting rules, take those out and see if that helps? Although, I’m guessing the error it is throwing could be caused by more than just one thing.
thanks for the reply. i don’t have any complex/custom query rules. for now, i’ve enabled the watchdog in poller settings which restarts the service every so often if the log file hasn’t been updated.
i can see short gaps in my graphs every so often when the service stops polling for a few min until the service is restarted by the watchdog. i don’t see the python stack trace at the same time anymore but something is still not quite right, will continue to debug when i get a chance.
