Regression in 26.9.1.1: dispatcher crashes

Danael_Braux · 29 September 2026 07:35

Steps to reproduce an issue

Environment: LibreNMS 26.9.1.1 (via the release update channel), Debian 13, PHP 8.4.24, Python 3.13.5, MariaDB 11.8.6 (separate DB server), RRDtool 1.7.2. Distributed polling with the dispatcher service, Redis for locks/queues, rrdcached on the main node reached over the network. 4 nodes in poller group 0: main (web, Redis, rrdcached, dispatcher) + 3 dispatcher-only pollers. rrd.step = 300 on all 4 nodes.
service_poller_frequency is left at its default null value (as documented in the Settings UI — see below).
Restart the dispatcher service on any node (systemctl restart librenms), or install a brand-new dispatcher-only node.
The service starts, then every worker crashes within seconds.

This is a regression, not a config issue: before updating to 26.9.1.1, distributed polling worked fine with service_poller_frequency at null. After the update, any node that restarts its dispatcher hits this crash — including my main node, restarted for unrelated maintenance. Two other nodes not yet restarted since the update kept working, confirming this triggers on dispatcher restart, not something node-specific.

The Settings UI itself documents that service_poller_frequency should stay null:

“How often to poll devices. Sets the default value for all nodes. Warning! This should normally be blank/null to match the rrd step option, otherwise you may break graphs. See docs for more info.”

What I found while investigating

In LibreNMS/service.py:

self.poller.frequency = config.get(
    "service_poller_frequency",
    config.get("rrd").get("step", ServiceConfig.poller.frequency),
)

dict.get(key, default) only returns the default when the key is absent. Since service_poller_frequency exists but is null, the fallback to rrd.step never triggers, and self.config.poller.frequency ends up None. That None then reaches:

service.py:416 and 425 — the performance and watchdog threads, via LibreNMS/__init__.py:244 (time() % seconds)
service.py:634 (fetch_immediate_device_list) — self.config.poller.frequency - 1
LibreNMS/__init__.py:602 (lock) — int(expiration) < 1, reached through queuemanager.py:422 → 689 → 245

All three error types share this one root cause.

Workaround I’m currently using

lnms config:set service_poller_frequency 300

(matching rrd.step, so actual polling frequency is unchanged). This is a shared, global setting — apply once, then restart every dispatcher to pick it up. Confirmed working across all 4 of my nodes.

Caveat: this hardcodes a value the settings page says should auto-track rrd.step. If rrd.step is ever changed later, this hardcoded value would silently desync — exactly what the warning is meant to prevent. The real fix belongs in service.py, treating a None/null config value the same as an absent key.

The output of ./validate.php (main node only, after applying the workaround):

===========================================
Component | Version
--------- | -------
LibreNMS  | 26.9.1 (2026-09-22T19:33:27+02:00)
DB Schema | 2026_09_16_120000_widen_ent_physical_rev_columns (403)
PHP       | 8.4.26
Python    | 3.13.5
Database  | MariaDB 11.8.6-MariaDB-0+deb13u1 from Debian
RRDTool   | 1.7.2
SNMP      | 5.9.4.pre2
===========================================

[OK]    Composer Version: 2.10.3
[OK]    Dependencies up-to-date.
[OK]    Database Connected
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    Database and column collations are correct
[OK]    Database schema correct
[OK]    MySQL and PHP time match
[OK]    Distributed Polling setting is enabled globally
[OK]    Connected to rrdcached
[OK]    Active pollers found
[FAIL]  Some dispatcher nodes did not check in recently
Inactive Nodes:
SRV-LNMS-P01
SRV-LNMS-P02
SRV-LNMS-P03
[OK]    Locks are functional
[OK]    Python wrapper cron entry is not present
[OK]    Redis is functional
[OK]    No errors found with polling frequencies.
[OK]    rrdtool version ok
[OK]    Connected to rrdcached

Pastebin output (clean crash traceback, main node restart before the workaround):

srv-lnms:~$ sudo journalctl -u librenms -f
sept. 29 09:23:58 srv-lnms librenms-service.py[62310]:   File "/opt/librenms/LibreNMS/service.py", line 574, in start
sept. 29 09:23:58 srv-lnms librenms-service.py[62310]:     devices = self.fetch_immediate_device_list()
sept. 29 09:23:58 srv-lnms librenms-service.py[62310]:   File "/opt/librenms/LibreNMS/service.py", line 634, in fetch_immediate_device_list
sept. 29 09:23:58 srv-lnms librenms-service.py[62310]:     poller_find_time = self.config.poller.frequency - 1
sept. 29 09:23:58 srv-lnms librenms-service.py[62310]:                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~
sept. 29 09:23:58 srv-lnms librenms-service.py[62310]: TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'
sept. 29 09:24:00 srv-lnms librenms-service.py[62310]: Alerting(INFO):Checking alerts
sept. 29 09:24:00 srv-lnms librenms-service.py[62310]: Billing(INFO):Calculating billing
sept. 29 09:24:01 srv-lnms librenms-service.py[62310]: Billing(INFO):Completed billing run for calculate in 0.21s
sept. 29 09:24:01 srv-lnms librenms-service.py[62310]: Alerting(INFO):Completed alerting run for alerts in 0.21s

===========================================
Component | Version
--------- | -------
LibreNMS  | 26.9.1 (2026-09-22T19:33:27+02:00)
DB Schema | 2026_09_16_120000_widen_ent_physical_rev_columns (403)
PHP       | 8.4.26
Python    | 3.13.5
Database  | MariaDB 11.8.6-MariaDB-0+deb13u1 from Debian
RRDTool   | 1.7.2
SNMP      | 5.9.4.pre2
===========================================

[OK]    Composer Version: 2.10.3
[OK]    Dependencies up-to-date.
[OK]    Database Connected
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    Database and column collations are correct
[OK]    Database schema correct
[OK]    MySQL and PHP time match
[OK]    Distributed Polling setting is enabled globally
[OK]    Connected to rrdcached
[FAIL]  The poller is not running. No poller ran in the last 300 seconds.
[FAIL]  No active dispatcher nodes found
[OK]    Locks are functional
[FAIL]  Python wrapper cron entry is not present
[OK]    Redis is functional
[OK]    No errors found with polling frequencies.
[OK]    rrdtool version ok
[OK]    Connected to rrdcached

Is this a regression introduced in the 26.9.x line, or is there a configuration step I’m missing that would have prevented this?

Thanks!

laf · 29 September 2026 08:31

I’m not saying the code doesn’t need hardening here to protect from any issues, however with service_poller_frequency to blank (lnms config:set service_poller_frequency) I’m not seeing a crash.

What was service_poller_frequency set to before for you?

Danael_Braux · 30 September 2026 11:49

It was always blank/null : I left it that way because of the UI warning, and distributed polling worked fine like that until I updated to 26.9.1.1. I applied the workaround (service_poller_frequency = 300) afterward.

Did I misread that warning?

laf · 30 September 2026 14:06

Yes but what was it actually set to, blank and null are not the same.

murrant · 30 September 2026 20:14

@eskyuu can you take a look?

Danael_Braux · 6 October 2026 08:20

According to the docs, i set via lnms config:set service_poller_frequency null. so it should be a literal NULL, not an empty string.