Daily.sh not running automatically

Tosti_ijzer · 28 August 2019 07:03

Hi all,

I hope this question has not been answered before and that this post is “yet another”. I am using LibreNMS on CentOS but for some reason my ./daily.sh gives me an error everyday. I am a novice with Linux so there will probably be some newbie questions or mistakes, sorry for that in advance.

Warning: Your install is over 24 hours out of date, last update: Mon, 26 Aug 2019 14:42:27 +0000
Fix:
Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.

My Cron jobs for LibreNMS:

[root@localhost user]# cat /etc/cron.d/librenms
#Using this cron file requires an additional user on your system, please see install docs.

33 */6 * * * librenms /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1
*/5 * * * * librenms /opt/librenms/discovery.php -h new >> /dev/null 2>&1
*/5 * * * * librenms /opt/librenms/cronic /opt/librenms/poller-wrapper.py 16
** * * * librenms /opt/librenms/alerts.php >> /dev/null 2>&1
*/5 * * * * librenms /opt/librenms/poll-billing.php >> /dev/null 2>&1
01 * * * * librenms /opt/librenms/billing-calculate.php >> /dev/null 2>&1
*/5 * * * * librenms /opt/librenms/check-services.php >> /dev/null 2>&1

#Daily maintenance script. DO NOT DISABLE!
#If you want to modify updates:
#Switch to monthly stable release: Choosing a release - LibreNMS Docs
#Disable updates: Updating - LibreNMS Docs
15 0 * * * librenms /opt/librenms/daily.sh >> /dev/null 2>&1

My ./validate.php tells me the following. Before you say, yeah you have to fix the local git modified files: I have a cron job running an extra script which I will post after the ./validate.php

./validate.php

Component Version

LibreNMS 1.54-56-g7783776

DB Schema 2019_07_09_150217_update_users_widgets_settings (140)

PHP 7.2.19

MySQL 5.5.60-MariaDB

RRDTool 1.4.8

SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 1.9.0
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[WARN] Your install is over 24 hours out of date, last update: Mon, 26 Aug 2019 14:42:27 +0000
[FIX]:
Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.
[WARN] Your local git contains modified files, this could prevent automatic updates.
[FIX]:
You can fix this with ./scripts/github-remove
Modified Files:
bootstrap/cache/.gitignore
logs/.gitignore
rrd/.gitignore
storage/app/.gitignore
storage/app/public/.gitignore
storage/debugbar/.gitignore
storage/framework/cache/.gitignore
storage/framework/cache/data/.gitignore
storage/framework/sessions/.gitignore
storage/framework/testing/.gitignore
storage/framework/views/.gitignore
storage/logs/.gitignore

Now, the reason there are modified files, I think at least, is because I created a ./fixrights.sh and added a cron for this so it runs every few minutes. The reason I created this is because LibreNMS was constantly losing user rights. I have checked the forum and found users with the same issue. They solved it with the script, and it worked for me too. Important notice: I did this because I thought it was the reason that the ./daily.sh script did not run every day. Not sure if I fixed it partially or not at al. In any case, the script does the following:

./fixrights.sh
sudo chown -R librenms:librenms /opt/librenms
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
sudo setfacl -R -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
chown -R librenms:librenms /opt/librenms

However, if I follow the the command to remove the modified Git files, my ./validate looks like this:

./validate.php

Component Version

LibreNMS 1.54-56-g7783776

DB Schema 2019_07_09_150217_update_users_widgets_settings (140)

PHP 7.2.19

MySQL 5.5.60-MariaDB

RRDTool 1.4.8

SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 1.9.0
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[WARN] Your install is over 24 hours out of date, last update: Mon, 26 Aug 2019 14:42:27 +0000
[FIX]:
Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.
[FAIL] We have found some files that are owned by a different user than librenms, this will stop you updating automatically and / or rrd files being updated causing graphs to fail.
[FIX]:
sudo chown -R librenms:librenms /opt/librenms
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
Files:
/opt/librenms/.git/index
/opt/librenms/logs/.gitignore
/opt/librenms/rrd/.gitignore
/opt/librenms/storage/app/public/.gitignore
/opt/librenms/storage/app/.gitignore
/opt/librenms/storage/framework/testing/.gitignore
/opt/librenms/storage/logs/.gitignore

Again, the script I have will fix the user rights because even if I enter the suggested commands to give ownership towards my Librenms user it will revert itself.

Does anyone have any idea what I am doing wrong?

If it helps, my ./daily.sh runs fine if I enter the command:

./daily.sh
Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase OK
Updating Composer packages OK
Updated from 7783776 to 7811227 OK
Updating SQL-Schema OK
Updating submodules OK
Cleaning up DB OK
Fetching notifications OK
Caching PeeringDB data OK

Aside from this, LibreNMS runs perfectly fine. I hope someone can help.

murrant · 28 August 2019 12:54

github-remove -d will fix the .gitignore files.
Sometimes when something hasn’t been merged that day you get that message from daily.sh.

If you want to help that, go test and review some pull requests on github

garysteers · 29 August 2019 19:35

Might be a silly question, but have you followed the guidance at the top of the page…

su - librenms
git pull
./daily.sh

Tosti_ijzer · 2 September 2019 09:46

Thanks for both your inputs! I will try those options.

K4S1 · 6 January 2020 13:37

Guess it’s fit here:

I’m getting the “Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.”

But not sure why. all seems up to date and good to go but keeps warning. any inputs ?

librenms@LNMS:~$ ./daily.sh 
Updating to latest codebase                        OK
Updating Composer packages                         OK
Updating SQL-Schema                                OK
Updating submodules                                OK
Cleaning up DB                                     OK
Fetching notifications                             OK
Caching PeeringDB data                             OK
librenms@LNMS:~$ ./validate.php 
====================================
Component | Version
--------- | -------
LibreNMS  | 1.59-531-gd9add2c
DB Schema | 2019_12_16_140000_create_customoids_table (152)
PHP       | 7.2.16-1+ubuntu16.04.1+deb.sury.org+1
MySQL     | 10.0.38-MariaDB-0ubuntu0.16.04.1
RRDTool   | 1.5.5
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.9.1
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[WARN]  Your install is over 24 hours out of date, last update: Sat, 04 Jan 2020 23:15:13 +0000
	[FIX]: 
	Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.
librenms@LNMS:~$ cat /etc/cron.d/librenms 
# Using this cron file requires an additional user on your system, please see install docs.

#33  */6   * * *   librenms    /opt/librenms/discovery.php -h all >> /dev/null 2>&1
33   */6  * * *   librenms    /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1
*/5  *    * * *   librenms    /opt/librenms/discovery.php -h new >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/cronic /opt/librenms/poller-wrapper.py 4
15   0    * * *   librenms    /opt/librenms/daily.sh >> /dev/null 2>&1
*    *    * * *   librenms    /opt/librenms/alerts.php >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/poll-billing.php >> /dev/null 2>&1
01   *    * * *   librenms    /opt/librenms/billing-calculate.php >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/check-services.php >> /dev/null 2>&1
*/2  *    * * *   librenms    /opt/librenms/html/plugins/Weathermap/map-poller.php >> /dev/null 2>&1
librenms@LNMS:~$ git pull
Already up-to-date.
librenms@LNMS:~$ 
librenms@LNMS:~$ ./daily.sh
Updating to latest codebase                        OK
Updating Composer packages                         OK
Updating SQL-Schema                                OK
Updating submodules                                OK
Cleaning up DB                                     OK
Fetching notifications                             OK
Caching PeeringDB data                             OK
librenms@LNMS:~$ ./validate.php 
====================================
Component | Version
--------- | -------
LibreNMS  | 1.59-531-gd9add2c
DB Schema | 2019_12_16_140000_create_customoids_table (152)
PHP       | 7.2.16-1+ubuntu16.04.1+deb.sury.org+1
MySQL     | 10.0.38-MariaDB-0ubuntu0.16.04.1
RRDTool   | 1.5.5
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.9.1
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[WARN]  Your install is over 24 hours out of date, last update: Sat, 04 Jan 2020 23:15:13 +0000
	[FIX]: 
	Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.
librenms@LNMS:~$

Chas · 6 January 2020 15:08

I think that logic for “Your install is over 24 hours out of date” checks the timezone of the system and compares it to the last git commit. If 1 day or more of difference it will throw that error. Since the last commit was 2 days ago i think it’s safe to ignore https://github.com/librenms/librenms/pulls?utf8=✓&q=is%3Amerged.

....
if ($commit_date->diff(new DateTime())->days > 0) {
                        $validator->warn(
                            "Your install is over 24 hours out of date
...

K4S1 · 7 January 2020 14:21

Arh okay now I understand. thanks for explaining to me.

See that there haven’t been done any updates for a coble of days:

Merged #11000 Correct ‘shortend’ misspelling 2 minutes ago
Merged #10916 Graphing Device Dependency 1 hour ago
Merged #10999 unix.inc.php: correct preg_match() pattern on line 5 3 days ago
Merged #10995 Add Blade directives for common Url functions 3 days ago

just did an update after I so the 2 commits and correct the warning disappeared:

librenms@LNMS:~$ date
Tue  7 Jan 15:04:47 CET 2020
librenms@nooslLNMS:~$ ./daily.sh 
Updating to latest codebase                        OK
Updating Composer packages                         OK
Updated from d9add2c to a65c9e8                    OK
Updating SQL-Schema                                OK
Updating submodules                                OK
Cleaning up DB                                     OK
Fetching notifications                             OK
Caching PeeringDB data                             OK
librenms@LNMS:~$ ./validate.php 
====================================
Component | Version
--------- | -------
LibreNMS  | 1.59-534-ga65c9e8
DB Schema | 2019_12_16_140000_create_customoids_table (152)
PHP       | 7.2.26-1+ubuntu16.04.1+deb.sury.org+1
MySQL     | 10.0.38-MariaDB-0ubuntu0.16.04.1
RRDTool   | 1.5.5
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.9.1
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
librenms@LNMS:~$

Component	Version
LibreNMS	1.54-56-g7783776
DB Schema	2019_07_09_150217_update_users_widgets_settings (140)
PHP	7.2.19
MySQL	5.5.60-MariaDB
RRDTool	1.4.8
SNMP	NET-SNMP 5.7.2
====================================