Device polling stopped working

Help
1

My LibreNMS instance was just fine. But today I found out that something happened to poller - it does not pull devices anymore. I got stuck in troublshooting

  1. I looked at cron job logs - poller wrapper runs every 5 min as expected.
  2. However if I run cron job manually i.e. [./cronic /opt/librenms/poller-wrapper.py 16] I recevie error - can’t connect to database: RESULT CODE: 2; STANDARD OUTPUT: ERROR: Could not connect to MySQL database!
  3. If I run poller manually - no problems.
  4. I have only about 50 devices so polling finishes in less than 200 seconds.
  5. I have not touched my instance for month - so no changes has been made (with exception to automatic updates)that could cause this issue

I don’t understand why poller-wrapper.py reports db connection error?

[root@librenms librenms]# ./validate.php

Component Version
LibreNMS 1.32-53-gee5c223
DB Schema 211
PHP 7.0.20
MySQL 5.5.52-MariaDB
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2

====================================

[OK] Database connection successful
[OK] Database schema correct
[FAIL] The poller has not run in the last 5 minutes, check the cron job
[WARN] Some devices have not been polled in the last 5 minutes.

########################################################################

[root@librenms librenms]# ./cronic /opt/librenms/poller-wrapper.py 16
Cronic detected failure or error output for the command:
/opt/librenms/poller-wrapper.py 16

RESULT CODE: 2

ERROR OUTPUT:

STANDARD OUTPUT:
ERROR: Could not connect to MySQL database!
[root@librenms librenms]#

#######################################################

[root@librenms librenms]# ./poller.php -h 6
LibreNMS Poller
Starting polling run:

Hostname: sje014-qa-3750g.illumio.net
Device ID: 6
OS: ios (cisco)

Uptime: 357 days, 6h 50m 36s
Module [ unix-agent ] disabled globally.

#######################################################

2

run ./daliy.sh

this was a bug that got through. :frowning:

then wait for the poller to cycle

3

great! Thank you Kevin - it worked.

1 Like
4

I experienced this as well. Looks like polling stopped last night around midnight EST.

pi@piLibreNMS:/opt/librenms $ sudo ./daily.sh
Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase OK
Updating SQL-Schema OK
Updating submodules OK
Cleaning up DB OK
Fetching notifications OK
Caching PeeringDB data OK
pi@piLibreNMS:/opt/librenms $ sudo ./validate.php

Component Version
LibreNMS 1.32-57-g163d0b4
DB Schema 211
PHP 7.0.19-1
MySQL 10.0.32-MariaDB-0+deb8u1
RRDTool 1.4.8
SNMP NET-SNMP 5.7.3

====================================

[OK] Database connection successful
[OK] Database schema correct
[FAIL] The poller has not run in the last 5 minutes, check the cron job
[WARN] Some devices have not been polled in the last 5 minutes.
You may have performance issues. Check your poll log and see: http://docs.librenms.org/Support/Performance/

I ran sudo ./daily.sh and sudo ./validate.php. I also double checked my cron job and it appears to be correct.

It does appear that the update has fixed the poller issue. Thank you, Kevin.

1 Like