The poller has not completed within the last 5 minutes, check the cron job

Hi everyone,

Slowly moving along with my Librenms install. I still can’t seem to fix this error.

validate.php output…

bash-4.4# ./validate.php

Component Version
LibreNMS 1.47
DB Schema 275
PHP 7.2.13
MySQL 5.5.56-MariaDB
RRDTool 1.7.0
SNMP NET-SNMP 5.7.3

====================================

[OK] Composer Version: 1.8.0
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[FAIL] The poller (8616b9cb2a58) has not completed within the last 5 minutes, check the cron job.
[FAIL] The poller (88ebe6420b94) has not completed within the last 5 minutes, check the cron job.
[FAIL] The poller (librenms) has not completed within the last 5 minutes, check the cron job.
[WARN] Some devices have not been polled in the last 5 minutes. You may have performance issues.
[FIX]:
Check your poll log and see: http://docs.librenms.org/Support/Performance/
Devices:
redacted
and 73 more…
[WARN] IPv6 is disabled on your server, you will not be able to add IPv6 devices.
[WARN] Your install is over 24 hours out of date, last update: Sun, 30 Dec 2018 14:29:16 +0000
[FIX]:
Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.
[WARN] Your local git branch is not master, this will prevent automatic updates.
[FIX]:
You can switch back to master with git checkout master
[WARN] Your local git contains modified files, this could prevent automatic updates.
[FIX]:
You can fix this with ./scripts/github-remove
Modified Files:
snmp-scan.py
[FAIL] Some folders have incorrect file permissions, this may cause issues.
[FIX]:
sudo chown -R librenms:librenms /opt/librenms
sudo setfacl -d -m g::rwx /data/rrd /data/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
sudo chmod -R ug=rwX /data/rrd /data/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
Files:
/opt/librenms/storage/framework/views/5fac4720fada9c5b0f8b760dcdca9a56c73755b2.php

Crontab…

bash-4.4# su librenms
~ $ crontab -l
# Using this cron file requires an additional user on your system, please see install docs.

33   */6  * * *   librenms    /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1
*/5  *    * * *   librenms    /opt/librenms/discovery.php -h new >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/cronic /opt/librenms/poller-wrapper.py 16
*    *    * * *   librenms    /opt/librenms/alerts.php >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/poll-billing.php >> /dev/null 2>&1
01   *    * * *   librenms    /opt/librenms/billing-calculate.php >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/check-services.php >> /dev/null 2>&1

# Daily maintenance script. DO NOT DISABLE!
# If you want to modify updates:
#  Switch to monthly stable release: https://docs.librenms.org/General/Releases/
#  Disable updates: https://docs.librenms.org/General/Updating/

15 0 * * * librenms /opt/librenms/daily.sh >> /dev/null 2>&1

~ $ ./poller-wrapper.py
INFO: starting the poller at 2019-01-31 15:48:58 with 16 threads, slowest devices first
INFO: worker Thread-3 finished device 7 in 20 seconds
INFO: worker Thread-2 finished device 6 in 20 seconds
INFO: worker Thread-13 finished device 44 in 21 seconds
INFO: worker Thread-7 finished device 67 in 23 seconds
INFO: worker Thread-10 finished device 66 in 23 seconds
INFO: worker Thread-16 finished device 48 in 23 seconds
INFO: worker Thread-8 finished device 49 in 23 seconds
INFO: worker Thread-14 finished device 79 in 23 seconds
INFO: worker Thread-15 finished device 82 in 23 seconds
INFO: worker Thread-9 finished device 54 in 24 seconds
INFO: worker Thread-6 finished device 37 in 25 seconds
INFO: worker Thread-11 finished device 43 in 26 seconds
INFO: worker Thread-12 finished device 28 in 34 seconds
INFO: worker Thread-5 finished device 14 in 38 seconds
INFO: worker Thread-4 finished device 15 in 38 seconds
INFO: worker Thread-3 finished device 85 in 23 seconds
INFO: worker Thread-2 finished device 52 in 23 seconds
INFO: worker Thread-1 finished device 5 in 45 seconds
INFO: worker Thread-7 finished device 84 in 25 seconds
INFO: worker Thread-10 finished device 83 in 25 seconds
INFO: worker Thread-8 finished device 80 in 25 seconds
INFO: worker Thread-14 finished device 13 in 25 seconds
INFO: worker Thread-16 finished device 81 in 25 seconds
INFO: worker Thread-15 finished device 23 in 25 seconds
INFO: worker Thread-13 finished device 38 in 27 seconds
INFO: worker Thread-6 finished device 22 in 24 seconds
INFO: worker Thread-9 finished device 20 in 25 seconds
INFO: worker Thread-11 finished device 50 in 24 seconds
INFO: worker Thread-12 finished device 34 in 27 seconds
INFO: worker Thread-5 finished device 53 in 23 seconds
INFO: worker Thread-4 finished device 41 in 24 seconds
INFO: worker Thread-3 finished device 51 in 24 seconds
INFO: worker Thread-11 finished device 8 in 17 seconds
INFO: worker Thread-2 finished device 42 in 24 seconds
INFO: worker Thread-1 finished device 21 in 23 seconds
INFO: worker Thread-7 finished device 39 in 23 seconds
INFO: worker Thread-10 finished device 19 in 23 seconds
INFO: worker Thread-8 finished device 40 in 23 seconds
INFO: worker Thread-14 finished device 18 in 23 seconds
INFO: worker Thread-15 finished device 26 in 23 seconds
INFO: worker Thread-9 finished device 36 in 23 seconds
INFO: worker Thread-13 finished device 27 in 24 seconds
INFO: worker Thread-6 finished device 35 in 24 seconds
INFO: worker Thread-5 finished device 9 in 17 seconds
INFO: worker Thread-12 finished device 10 in 17 seconds
INFO: worker Thread-4 finished device 25 in 17 seconds
INFO: worker Thread-16 finished device 30 in 34 seconds
INFO: worker Thread-3 finished device 24 in 17 seconds
INFO: worker Thread-2 finished device 11 in 17 seconds
INFO: worker Thread-1 finished device 12 in 17 seconds
INFO: worker Thread-7 finished device 33 in 23 seconds
INFO: worker Thread-10 finished device 32 in 23 seconds
INFO: worker Thread-14 finished device 31 in 23 seconds
INFO: worker Thread-8 finished device 45 in 24 seconds
INFO: worker Thread-6 finished device 76 in 24 seconds
INFO: worker Thread-13 finished device 3 in 31 seconds
INFO: worker Thread-5 finished device 75 in 25 seconds
INFO: worker Thread-12 finished device 74 in 25 seconds
INFO: worker Thread-4 finished device 73 in 25 seconds
INFO: worker Thread-15 finished device 2 in 36 seconds
INFO: worker Thread-16 finished device 77 in 26 seconds
INFO: worker Thread-3 finished device 78 in 25 seconds
INFO: worker Thread-1 finished device 17 in 26 seconds
INFO: worker Thread-11 finished device 29 in 44 seconds
INFO: worker Thread-2 finished device 47 in 27 seconds
INFO: worker Thread-9 finished device 1 in 40 seconds
INFO: worker Thread-7 finished device 16 in 25 seconds
INFO: worker Thread-14 finished device 86 in 26 seconds
INFO: worker Thread-8 finished device 87 in 26 seconds
INFO: worker Thread-6 finished device 72 in 26 seconds
INFO: worker Thread-13 finished device 71 in 27 seconds
INFO: worker Thread-12 finished device 56 in 27 seconds
INFO: worker Thread-5 finished device 55 in 27 seconds
INFO: worker Thread-4 finished device 57 in 27 seconds
INFO: worker Thread-15 finished device 58 in 26 seconds
INFO: worker Thread-10 finished device 4 in 40 seconds
INFO: worker Thread-3 finished device 60 in 26 seconds
INFO: worker Thread-16 finished device 59 in 27 seconds
INFO: worker Thread-1 finished device 61 in 25 seconds
INFO: worker Thread-11 finished device 62 in 25 seconds
INFO: worker Thread-2 finished device 63 in 25 seconds
INFO: worker Thread-9 finished device 64 in 25 seconds
INFO: worker Thread-7 finished device 65 in 21 seconds
INFO: worker Thread-14 finished device 46 in 19 seconds
INFO: worker Thread-6 finished device 69 in 16 seconds
INFO: worker Thread-8 finished device 68 in 19 seconds
INFO: worker Thread-13 finished device 70 in 11 seconds
INFO: worker Thread-12 finished device 88 in 13 seconds
INFO: poller-wrapper polled 88 devices in 146 seconds with 16 workers

This is running on the Alpine/Nginix docker.

Is there something wrong with my cronjobs?

Do you have multiply pollers setup? If not you need to delete these.

Also run the commands this telling you to do.

Thank you for the response Kevin.  

I'll correct the other issues.   I had to change the snmp-scan.py to use python3 as it wouldn't work with python2.

I don't have multiple crons.   

~ $ crontab -l
# Using this cron file requires an additional user on your system, please see install docs.
#
*/6  *    * * *   librenms    /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1
*/5  *    * * *   librenms    /opt/librenms/discovery.php -h new >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/cronic /opt/librenms/poller-wrapper.py 16
*    *    * * *   librenms    /opt/librenms/alerts.php >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/poll-billing.php >> /dev/null 2>&1
01   *    * * *   librenms    /opt/librenms/billing-calculate.php >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/check-services.php >> /dev/null 2>&1

root crons..

bash-4.4# crontab -l
# do daily/weekly/monthly maintenance
# min	hour	day	month	weekday	command
*/15	*	*	*	*	run-parts /etc/periodic/15min
0	*	*	*	*	run-parts /etc/periodic/hourly
0	2	*	*	*	run-parts /etc/periodic/daily
0	3	*	*	6	run-parts /etc/periodic/weekly
0	5	1	*	*	run-parts /etc/periodic/monthly

I don't have any other users on the system that have shells.  

bash-4.4# ls /var/spool/cron/crontabs/
librenms  root
bash-4.4#
1 Like

Ok then you need to remove those pollers. In the web ui click on the gear icon --> Pollers --> Pollers. Now delete those extra pollers.

so I removed those extraneous pollers, thanks for the tip on that.

But my main poller is still complaining…

bash-4.4# ./validate.php

Component Version
LibreNMS 1.47
DB Schema 275
PHP 7.2.13
MySQL 5.5.56-MariaDB
RRDTool 1.7.0
SNMP NET-SNMP 5.7.3

====================================

[OK] Composer Version: 1.8.0
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[FAIL] The poller (librenms) has not completed within the last 5 minutes, check the cron job.
[WARN] Some devices have not been polled in the last 5 minutes. You may have performance issues.
[FIX]:

So any idea why the cron job isn’t working properly?

where are the cron jobs located? they should be in /etc/cron.d/librenms

There is no /etc/cron.d directory. Crons are apparently stored in /etc/crontabs/librenms in Alpine.

I can create that directory and put the crons there and see what happens.

Same problem.

Run top or htop see if the poller is running

You mean running persistently or being fired off as a cron?

Doesn’t look it is running either way. I can run it from command line though.

Annnnddd I figured out the problem. So I had two problems really. One of them container related it would seem and one of the Alpine Linux related.

So the container related issue is that I launced crond directly which didn’t seem to actually do anything. The process was running but kinda sitting in limbo. I actually had to run /usr/local/bin/cron which created a busyboxy cron process. A definitely RTFM issue on my end as the the docs say to run cron as a side-car.

The other problem was the structure of the crons. The default script has the cron structure as this…

33 */6 * * * librenms /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1

the librenms part was breaking the script.

2 Likes