Cron - 6 hours discovery not working

Hi,

after i jumped to the 21.10.x release (I am using the montly releases only)
i noticed an issue in the 6 hours discovery process of my devices.
I am using the cron based polling and discovery (not the dispatcher service)
Right now I am on [21.10.0-38-gbc8efedc8 - Thu Oct 21 2021 06:20:57 GMT+0200]
but discovery is not working at the moment either
My devices showing:

Last Discovered:

4 days 16 hours 6 minutes 5 seconds ago

Here is my setup:

====================================

Component Version
LibreNMS 21.10.0-38-gbc8efedc8
DB Schema 2021_10_20_224207_increase_length_of_attrib_type_column (224)
PHP 7.4.3
Python 3.8.10
MySQL 10.3.31-MariaDB-0ubuntu0.20.04.1
RRDTool 1.7.2
SNMP NET-SNMP 5.8

Maybe someone else who also noticed the same problem?

Thanks and ciao.

What do your cron entries for discovery look like ?

My entries look like this:

33   */6  * * *   librenms    /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 2
*/5  *    * * *   librenms    /opt/librenms/discovery.php -h new >> /dev/null 2>&1

The first one runs a full discovery scan every 6 hours using the threaded discovery-wrapper, the second runs a discovery of only new devices every 5 minutes using discovery.php.

What happens if you run discovery manually ?

For example using the threaded wrapper. First switch to the librenms user whatever that is:

sudo su - librenms

Then

/opt/librenms/discovery-wrapper.py 2

You should see something like:

2021-10-21 11:14:31,493 :: INFO :: starting the discovery check at 2021-10-21 11:14:31 with 2 threads for 340 devices
2021-10-21 11:15:18,583 :: INFO :: worker Thread-2 finished device 274 in 47 seconds
2021-10-21 11:15:22,069 :: INFO :: worker Thread-1 finished device 169 in 50 seconds
2021-10-21 11:16:11,770 :: INFO :: worker Thread-1 finished device 14 in 49 seconds

Or if you want more details you can use the single threaded discovery.php in “all” mode:

/opt/librenms/discovery.php -h all

This is slow but will give a very verbose output and hopefully pinpoint any errors you might be seeing.

I double checked my cron entries. they are correct.
I will check discovery.php -h all

thanks

manually /opt/librenms/discovery-wrapper.py 2
seems to work. Somehow it seems via cron it doesn’t work

my cron tab entry is:
33 */6 * * * librenms /opt/librenms/cronic /opt/librenms/discovery-wrapper.py
*/5 * * * * librenms /opt/librenms/discovery.php -h new >> /dev/null 2>&1

But i didn’t change anything there anyway.
hmm. weird.

Does it still work if you run it manually at the command line through cronic ? eg:

/opt/librenms/cronic /opt/librenms/discovery-wrapper.py

If it does it’s something wrong with your cron configuration or environment. Not sure what though.

it hangs when I do:

[email protected]:~$ /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 2

->>> nothing happens here. it seems to hang

Give it plenty of time to complete. Cronic is just a wrapper script that suppresses output unless there are errors, so it will appear to “hang” only because it doesn’t show any output. Cronic is there to reduce the number of emails sent by cron.

If you don’t have cron configured to send emails (or the emails go nowhere) you could just remove /opt/librenms/cronic from your cron line as it’s not necessary for it to work.

I have this MAILTO="" in my cron file.
A long time ago when i configured it. I guess it was needed to send emails or is it just
needed to send emails if cron has any problems?

By default cron will send an email for any script/command that has any output at all - good or bad. This can lead to a lot of annoying emails.

I don’t use cronic elsewhere but from looking at the script it appears to simply suppress output unless the wrapped script returns an error, then it passes it through so it can be emailed. That way only errors are emailed to you.

Another option is to simply redirect output to /dev/null, then you will never get an email from that script.

If you have MAILTO="" that probably suppresses all emails from cron, in that case just try removing /opt/librenms/cronic and run the discovery wrapper directly to see if that works.

Thanks for the infos.
It is real strange.
when i manually do:
/opt/librenms/cronic /opt/librenms/discovery-wrapper.py 16
I see in the log file: which is correct

/opt/librenms/discovery.php 1007 2021-10-21 13:54:48 - 1 devices discovered in 20.41 secs
/opt/librenms/poller.php 873 2021-10-21 13:54:48 - 1 devices polled in 10.37 secs
/opt/librenms/discovery.php 1006 2021-10-21 13:54:48 - 1 devices discovered in 20.60 secs

but when I do it via cron, it doesnt seem to work and no discovery log entries are created in the log
even when I remove the cronic .

Seems it works again now. Thanks for you hints and help.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.