FAIL: Discovery has not completed in the last 24 hours - Discovery-wrapper.py

I have read and tried a few different things I have found on the forums and yet I still have this issue with Discovery has not completed…

Here is some info:
CENTOS 7 / Nginx
I follow the documentation for the installation here: https://docs.librenms.org/Installation/Installation-CentOS-7-Nginx/

Validate as the libreNMS user:

====================================
Component | Version
--------- | -------
LibreNMS  | 1.47-51-gf8cf6a9
DB Schema | 1000
PHP       | 7.2.10
MySQL     | 5.5.60-MariaDB
RRDTool   | 1.4.8
SNMP      | NET-SNMP 5.7.2
====================================

[OK]    Composer Version: 1.8.0
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[FAIL]  Discovery has not completed in the last 24 hours.
	[FIX]:
	Check the cron job to make sure it is running and using discovery-wrapper.py

Cron log:

[root@localhost cron.d]# cat /var/log/cron | grep discovery-wrapper.py
Jan 13 06:33:01 localhost CROND[18518]: (librenms) CMD (   /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1)
Jan 13 12:33:01 localhost CROND[24099]: (librenms) CMD (   /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1)
Jan 13 18:33:01 localhost CROND[29575]: (librenms) CMD (   /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1)
Jan 14 00:33:01 localhost CROND[2918]: (librenms) CMD (   /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1)
Jan 14 06:33:01 localhost CROND[8498]: (librenms) CMD (   /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1)
Jan 14 12:33:01 localhost CROND[27225]: (librenms) CMD (   /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1)
Jan 14 18:33:01 localhost CROND[5192]: (librenms) CMD (   /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1)
Jan 15 00:33:01 localhost CROND[13574]: (librenms) CMD (   /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1)
Jan 15 06:33:01 localhost CROND[21716]: (librenms) CMD (   /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1)
[root@localhost librenms]# ./discovery.php -h HOSTNAME -d | ./pbin.sh
Paste endpoint: https://p.libren.ms/api/create
title:
name: root
private: 1
expire: 0
https://p.libren.ms/view/2acf65d1
[root@localhost librenms]# ./poller.php -h HOSTNAME -r -f -d | ./pbin.sh
Paste endpoint: https://p.libren.ms/api/create
title:
name: root
private: 1
expire: 0
https://p.libren.ms/view/f189f1f7

and the crontab:

cat /opt/librenms/librenms.nonroot.cron
# Using this cron file requires an additional user on your system, please see install docs.

33   */6  * * *   librenms    /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1
*/5  *    * * *   librenms    /opt/librenms/discovery.php -h new >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/cronic /opt/librenms/poller-wrapper.py 16
*    *    * * *   librenms    /opt/librenms/alerts.php >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/poll-billing.php >> /dev/null 2>&1
01   *    * * *   librenms    /opt/librenms/billing-calculate.php >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/check-services.php >> /dev/null 2>&1

# Daily maintenance script. DO NOT DISABLE!
# If you want to modify updates:
#  Switch to monthly stable release: https://docs.librenms.org/General/Releases/
#  Disable updates: https://docs.librenms.org/General/Updating/
15   0    * * *   librenms    /opt/librenms/daily.sh >> /dev/null 2>&1

its possible that it has not completed in the last 24 hours… But its also possible it has. Check the log files.

What? Do you need more information? This server has been running for over a year… the LibreNMS data shows data for traffic of my switches and such for over a year. I have had this problem for a year. I’m finally trying to once and for all fix it.

EDIT:
WAIT A SECOND YOU EDITTED YOUR RESPONSE SINCE THIS POST.

Where are the log files in question that you would like me to check?
Also if it has completed in the last 24 hours why does it fail everytime? Is this supposed to happen?

From Today:

bash-4.2$ ./validate.php
====================================
Component | Version
--------- | -------
LibreNMS  | 1.47-53-g8e80121
DB Schema | 1000
PHP       | 7.2.10
MySQL     | 5.5.60-MariaDB
RRDTool   | 1.4.8
SNMP      | NET-SNMP 5.7.2
====================================

[OK]    Composer Version: 1.8.0
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[FAIL]  Discovery has not completed in the last 24 hours.
	[FIX]:
	Check the cron job to make sure it is running and using discovery-wrapper.py

EDIT 2:
This is from the /opt/librenms/log/daily.log

Returned: 0
Cleaning up DB
Refreshing alert rules queries
Clearing OS cache
Eventlog cleared for entries over 30 days
Perf times cleared for entries over 30 days
Device perf cleared for entries over 7 days
Returned: 0
Fetching notifications
[ Wed, 16 Jan 2019 00:15:23 -0800 ] http://www.librenms.org/notifications.rss (39)
[ Wed, 16 Jan 2019 00:15:23 -0800 ] misc/notifications.rss (46)
[ Wed, 16 Jan 2019 00:15:23 -0800 ] Updating DB  Done
Returned: 0
Caching PeeringDB data
Peering DB integration disabled
Returned: 0

And from the libre.log greping for discovery (truncated)

/opt/librenms/discovery.php new 2019-01-16 07:30:01 - 0 devices discovered in 0.000 secs
/opt/librenms/discovery.php new 2019-01-16 07:35:02 - 0 devices discovered in 0.000 secs
/opt/librenms/discovery.php new 2019-01-16 07:40:01 - 0 devices discovered in 0.000 secs

or just a tail:

/opt/librenms/poller.php 32 2019-01-16 07:45:16 - 1 devices polled in 3.330 secs
/opt/librenms/poller.php 30 2019-01-16 07:45:16 - 1 devices polled in 3.126 secs
/opt/librenms/poller.php 7 2019-01-16 07:45:17 - 1 devices polled in 3.174 secs
/opt/librenms/poller.php 2 2019-01-16 07:45:18 - 1 devices polled in 1.743 secs
/opt/librenms/poller.php 19 2019-01-16 07:45:18 - 1 devices polled in 1.176 secs
/opt/librenms/poller.php 29 2019-01-16 07:45:19 - 1 devices polled in 3.171 secs
/opt/librenms/poller.php 27 2019-01-16 07:45:19 - 1 devices polled in 3.277 secs
/opt/librenms/poller.php 28 2019-01-16 07:45:19 - 1 devices polled in 3.250 secs
/opt/librenms/poller.php 16 2019-01-16 07:45:33 - 1 devices polled in 31.01 secs
/opt/librenms/poller.php 14 2019-01-16 07:45:33 - 1 devices polled in 30.96 secs

Can anyone please help? I’m more then willing to provide any other log info you need all you have to do is ask.
I have updated my last response with 2 edits including some log files.

I have ran “./discovery.php -v -h all” on my own and that seems to fix it, for now.
Although it took a good 871.0 seconds to complete. Here it is in the librenms.log:

./discovery.php all 2019-01-16 13:41:39 - 28 devices discovered in 871.0 secs

And here is the output of the ./validate:

./validate.php
====================================
Component | Version
--------- | -------
LibreNMS  | 1.47-53-g8e80121
DB Schema | 1000
PHP       | 7.2.10
MySQL     | 5.5.60-MariaDB
RRDTool   | 1.4.8
SNMP      | NET-SNMP 5.7.2
====================================

[OK]    Composer Version: 1.8.0
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct

This makes me think that it is not running. I don’t know the difference between discovery-wrapper.py and discovery.php … I will research this I guess. Unless anyone can answer this for me :slight_smile:

I am getting the same error. Any luck with this? I am checking my total discovery time now.