Daily update breaks poller

Time of incident: 2020-11-23 0:15 PST
Behavior: Poller is not successfully polling devices
OS: Debian GNU/Linux 9.13 (stretch)

discovery-wrapper and poller-wrapper reporting:
2020-11-23 16:35:01,461 :: ERROR :: Command [[’/usr/bin/env’, ‘php’, ‘/opt/librenms/config_to_json.php’]] failed for unknown reasons [init() got an unexpected keyword argument ‘errors’].
2020-11-23 16:35:01,461 :: ERROR :: ERROR: Could not execute: [’/usr/bin/env’, ‘php’, ‘/opt/librenms/config_to_json.php’]
2020-11-23 16:35:01,462 :: ERROR :: exit code: None. Output:
init() got an unexpected keyword argument ‘errors’

1 Like

Yup, I’m seeing this same broken behavior.

same situation, poller-wrapper.py stop from midnight.

We have the same situation.

Yep, same thing happened here.

Happened to us too.

Same problem here.
Anyone got a fix for it?

my temporary workaround :

edit /etc/cron.d/librenms

comment
*/5 * * * * librenms /opt/librenms/cronic /opt/librenms/poller-wrapper.py

add
*/5 * * * * librenms /opt/librenms/poller.php -h all

2 Likes

Same problem.

2020-11-24 11:10:01,571 :: ERROR :: Command [[’/usr/bin/env’, ‘php’, ‘/opt/librenms/config_to_json.php’]] failed for unknown reasons [init() got an unexpected keyword argument ‘encoding$
2020-11-24 11:10:01,571 :: ERROR :: ERROR: Could not execute: [’/usr/bin/env’, ‘php’, ‘/opt/librenms/config_to_json.php’]
2020-11-24 11:10:01,572 :: ERROR :: exit code: None. Output:
init() got an unexpected keyword argument ‘encoding’

1 Like

It seems to be a problem here

It seems Python 3.5 on Debian Stretch does not support errors or encoding value.

https://docs.python.org/3.5/library/subprocess.html#subprocess.check_output

Thus it fails.

Same here… After daily.sh this night poller stoped working, everything else works OK.

$ /opt/librenms/cronic /opt/librenms/poller-wrapper.py 16
Cronic detected failure or error output for the command:
/opt/librenms/poller-wrapper.py 16

RESULT CODE: 1

ERROR OUTPUT:
Traceback (most recent call last):
  File "/opt/librenms/poller-wrapper.py", line 28, in <module>
    import LibreNMS.library as LNMS
  File "/opt/librenms/LibreNMS/__init__.py", line 12, in <module>
    from .service import Service, ServiceConfig
  File "/opt/librenms/LibreNMS/service.py", line 6, in <module>
    import pymysql
ModuleNotFoundError: No module named 'pymysql'

STANDARD OUTPUT:

and also after apt install python3-pymysql…

$ /opt/librenms/cronic /opt/librenms/poller-wrapper.py 16
Cronic detected failure or error output for the command:
/opt/librenms/poller-wrapper.py 16

RESULT CODE: 1

ERROR OUTPUT:
Traceback (most recent call last):
  File "/opt/librenms/poller-wrapper.py", line 181, in <module>
    config = json.loads(LNMS.get_config_data(install_dir))
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

STANDARD OUTPUT:
$

Validate info…

$ /opt/librenms/validate.php
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
====================================
Component | Version
--------- | -------
LibreNMS  | 1.69-56-g305f97161
DB Schema | 2020_10_12_095504_mempools_add_oids (191)
PHP       | 7.4.3
Python    | 3.8.5
MySQL     | 10.3.25-MariaDB-0ubuntu0.20.04.1
RRDTool   | 1.7.2
SNMP      | NET-SNMP 5.8
OpenSSL   | 
====================================

[OK]    Composer Version: 2.0.7
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
$ uname -a
Linux librenms 5.4.0-54-generic #60-Ubuntu SMP Fri Nov 6 10:37:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$

For Debian Stretch I could fix it with:

diff --git a/LibreNMS/library.py b/LibreNMS/library.py
index cb24a691d..fe3e607d9 100644
--- a/LibreNMS/library.py
+++ b/LibreNMS/library.py
@@ -139,8 +139,8 @@ def command_runner(command, valid_exit_codes=None, timeout=300, shell=False, enc
         # Disabling pylint error for the same reason as above
         # pylint: disable=E1123
         output = subprocess.check_output(command, stderr=subprocess.STDOUT, shell=shell,
-                                         timeout=timeout, universal_newlines=universal_newlines, encoding=encoding,
-                                         errors=errors, creationflags=creationflags, **kwargs)
+                                         timeout=timeout, universal_newlines=universal_newlines,
+                                         creationflags=creationflags, **kwargs).decode(encoding)
 
     except subprocess.CalledProcessError as exc:
         exit_code = exc.returncode

FYI, till todays daily.sh poller was working (dont know why now it needs pymysql and before not).
If I start poller via GUI/capture, it will poll the device without problem.
All other cron scripts are running OK (like discovery, alerts, etc).

Also added a comment in the code and the github Issue:

1 Like

FYI, to git checkout 4bdcb7d88e0d92ad507ca39215e2fc04231bffea (before “Python log mysql connection errors and harden config retrieval” commit) and it will work again. And disable daily.sh so it wont broke again till it’s fixed :wink:

1 Like

change reverted you can run ./daily.sh

4 Likes

I’ve tested it, it’s OK now and puller is working as expected. Thx @murrant :slight_smile:

Can also confirmed to be working! Many thanks!

I got different errors too:

[2020-11-24 00:15:17] production.ERROR: Call to undefined method LibreNMS\Util\Rewrite::formatStorage() {"exception":"[object] (Error(code: 0): Call to undefined method LibreNMS\\Util\\Rewrite::formatStorage() at /opt/librenms/includes/common.php:492)
[stacktrace]
#0 /opt/librenms/includes/polling/ports.inc.php(791): formatStorage()
#1 /opt/librenms/includes/polling/functions.inc.php(317): include('/opt/librenms/i...')
#2 /opt/librenms/poller.php(140): poll_device()
#3 {main}
"}

and the poller stopped working.

ran again .daily.sh few minutes ago and it works again.