Daily update breaks poller

[Morning_Raktajino]

Time of incident: 2020-11-23 0:15 PST
Behavior: Poller is not successfully polling devices
OS: Debian GNU/Linux 9.13 (stretch)

discovery-wrapper and poller-wrapper reporting:
2020-11-23 16:35:01,461 :: ERROR :: Command [[’/usr/bin/env’, ‘php’, ‘/opt/librenms/config_to_json.php’]] failed for unknown reasons [init() got an unexpected keyword argument ‘errors’].
2020-11-23 16:35:01,461 :: ERROR :: ERROR: Could not execute: [’/usr/bin/env’, ‘php’, ‘/opt/librenms/config_to_json.php’]
2020-11-23 16:35:01,462 :: ERROR :: exit code: None. Output:
init() got an unexpected keyword argument ‘errors’

[ars]

Yup, I’m seeing this same broken behavior.

[chianan]

same situation, poller-wrapper.py stop from midnight.

[Dmitry_Chepurnenko]

We have the same situation.

[awlx]

Yep, same thing happened here.

[Joarc]

Happened to us too.

[Mikael_Olsson]

Same problem here.
Anyone got a fix for it?

[chianan]

my temporary workaround :

edit /etc/cron.d/librenms

comment
*/5 * * * * librenms /opt/librenms/cronic /opt/librenms/poller-wrapper.py

add
*/5 * * * * librenms /opt/librenms/poller.php -h all

[freedomwarrior]

Same problem.

2020-11-24 11:10:01,571 :: ERROR :: Command [[‘/usr/bin/env’, ‘php’, ‘/opt/librenms/config_to_json.php’]] failed for unknown reasons [init() got an unexpected keyword argument ‘encoding$
2020-11-24 11:10:01,571 :: ERROR :: ERROR: Could not execute: [’/usr/bin/env’, ‘php’, ‘/opt/librenms/config_to_json.php’]
2020-11-24 11:10:01,572 :: ERROR :: exit code: None. Output:
init() got an unexpected keyword argument ‘encoding’

[awlx]

Python log mysql connection errors and harden config retrieval (#12058)

It seems to be a problem here

[awlx]

It seems Python 3.5 on Debian Stretch does not support errors or encoding value.

https://docs.python.org/3.5/library/subprocess.html#subprocess.check_output

Thus it fails.

[oszafraniec]

Same here… After daily.sh this night poller stoped working, everything else works OK.

$ /opt/librenms/cronic /opt/librenms/poller-wrapper.py 16
Cronic detected failure or error output for the command:
/opt/librenms/poller-wrapper.py 16

RESULT CODE: 1

ERROR OUTPUT:
Traceback (most recent call last):
  File "/opt/librenms/poller-wrapper.py", line 28, in <module>
    import LibreNMS.library as LNMS
  File "/opt/librenms/LibreNMS/__init__.py", line 12, in <module>
    from .service import Service, ServiceConfig
  File "/opt/librenms/LibreNMS/service.py", line 6, in <module>
    import pymysql
ModuleNotFoundError: No module named 'pymysql'

STANDARD OUTPUT:

and also after apt install python3-pymysql…

$ /opt/librenms/cronic /opt/librenms/poller-wrapper.py 16
Cronic detected failure or error output for the command:
/opt/librenms/poller-wrapper.py 16

RESULT CODE: 1

ERROR OUTPUT:
Traceback (most recent call last):
  File "/opt/librenms/poller-wrapper.py", line 181, in <module>
    config = json.loads(LNMS.get_config_data(install_dir))
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

STANDARD OUTPUT:
$

Validate info…

$ /opt/librenms/validate.php
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
====================================
Component | Version
--------- | -------
LibreNMS  | 1.69-56-g305f97161
DB Schema | 2020_10_12_095504_mempools_add_oids (191)
PHP       | 7.4.3
Python    | 3.8.5
MySQL     | 10.3.25-MariaDB-0ubuntu0.20.04.1
RRDTool   | 1.7.2
SNMP      | NET-SNMP 5.8
OpenSSL   | 
====================================

[OK]    Composer Version: 2.0.7
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
$ uname -a
Linux librenms 5.4.0-54-generic #60-Ubuntu SMP Fri Nov 6 10:37:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$

[awlx]

For Debian Stretch I could fix it with:

diff --git a/LibreNMS/library.py b/LibreNMS/library.py
index cb24a691d..fe3e607d9 100644
--- a/LibreNMS/library.py
+++ b/LibreNMS/library.py
@@ -139,8 +139,8 @@ def command_runner(command, valid_exit_codes=None, timeout=300, shell=False, enc
         # Disabling pylint error for the same reason as above
         # pylint: disable=E1123
         output = subprocess.check_output(command, stderr=subprocess.STDOUT, shell=shell,
-                                         timeout=timeout, universal_newlines=universal_newlines, encoding=encoding,
-                                         errors=errors, creationflags=creationflags, **kwargs)
+                                         timeout=timeout, universal_newlines=universal_newlines,
+                                         creationflags=creationflags, **kwargs).decode(encoding)
 
     except subprocess.CalledProcessError as exc:
         exit_code = exc.returncode

[oszafraniec]

FYI, till todays daily.sh poller was working (dont know why now it needs pymysql and before not).
If I start poller via GUI/capture, it will poll the device without problem.
All other cron scripts are running OK (like discovery, alerts, etc).

[awlx]

Also added a comment in the code and the github Issue:

Poller.php fail to run

[oszafraniec]

FYI, to git checkout 4bdcb7d88e0d92ad507ca39215e2fc04231bffea (before “Python log mysql connection errors and harden config retrieval” commit) and it will work again. And disable daily.sh so it wont broke again till it’s fixed

[murrant]

change reverted you can run ./daily.sh

[oszafraniec]

I’ve tested it, it’s OK now and puller is working as expected. Thx @murrant

[Joarc]

Can also confirmed to be working! Many thanks!

[exeral]

I got different errors too:

[2020-11-24 00:15:17] production.ERROR: Call to undefined method LibreNMS\Util\Rewrite::formatStorage() {"exception":"[object] (Error(code: 0): Call to undefined method LibreNMS\\Util\\Rewrite::formatStorage() at /opt/librenms/includes/common.php:492)
[stacktrace]
#0 /opt/librenms/includes/polling/ports.inc.php(791): formatStorage()
#1 /opt/librenms/includes/polling/functions.inc.php(317): include('/opt/librenms/i...')
#2 /opt/librenms/poller.php(140): poll_device()
#3 {main}
"}

and the poller stopped working.

ran again .daily.sh few minutes ago and it works again.