Daily.sh not running automatically

Hi all,

I hope this question has not been answered before and that this post is “yet another”. I am using LibreNMS on CentOS but for some reason my ./daily.sh gives me an error everyday. I am a novice with Linux so there will probably be some newbie questions or mistakes, sorry for that in advance.

Warning: Your install is over 24 hours out of date, last update: Mon, 26 Aug 2019 14:42:27 +0000

Fix:
Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.

My Cron jobs for LibreNMS:

[[email protected] user]# cat /etc/cron.d/librenms
#Using this cron file requires an additional user on your system, please see install docs.

33 */6 * * * librenms /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1
*/5 * * * * librenms /opt/librenms/discovery.php -h new >> /dev/null 2>&1
*/5 * * * * librenms /opt/librenms/cronic /opt/librenms/poller-wrapper.py 16
** * * * librenms /opt/librenms/alerts.php >> /dev/null 2>&1
*/5 * * * * librenms /opt/librenms/poll-billing.php >> /dev/null 2>&1
01 * * * * librenms /opt/librenms/billing-calculate.php >> /dev/null 2>&1
*/5 * * * * librenms /opt/librenms/check-services.php >> /dev/null 2>&1

#Daily maintenance script. DO NOT DISABLE!
#If you want to modify updates:
#Switch to monthly stable release: https://docs.librenms.org/General/Releases/
#Disable updates: https://docs.librenms.org/General/Updating/
15 0 * * * librenms /opt/librenms/daily.sh >> /dev/null 2>&1

My ./validate.php tells me the following. Before you say, yeah you have to fix the local git modified files: I have a cron job running an extra script which I will post after the ./validate.php

./validate.php

Component Version
LibreNMS 1.54-56-g7783776
DB Schema 2019_07_09_150217_update_users_widgets_settings (140)
PHP 7.2.19
MySQL 5.5.60-MariaDB
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 1.9.0
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[WARN] Your install is over 24 hours out of date, last update: Mon, 26 Aug 2019 14:42:27 +0000
[FIX]:
Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.
[WARN] Your local git contains modified files, this could prevent automatic updates.
[FIX]:
You can fix this with ./scripts/github-remove
Modified Files:
bootstrap/cache/.gitignore
logs/.gitignore
rrd/.gitignore
storage/app/.gitignore
storage/app/public/.gitignore
storage/debugbar/.gitignore
storage/framework/cache/.gitignore
storage/framework/cache/data/.gitignore
storage/framework/sessions/.gitignore
storage/framework/testing/.gitignore
storage/framework/views/.gitignore
storage/logs/.gitignore

Now, the reason there are modified files, I think at least, is because I created a ./fixrights.sh and added a cron for this so it runs every few minutes. The reason I created this is because LibreNMS was constantly losing user rights. I have checked the forum and found users with the same issue. They solved it with the script, and it worked for me too. Important notice: I did this because I thought it was the reason that the ./daily.sh script did not run every day. Not sure if I fixed it partially or not at al. In any case, the script does the following:

./fixrights.sh
sudo chown -R librenms:librenms /opt/librenms
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
sudo setfacl -R -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
chown -R librenms:librenms /opt/librenms

However, if I follow the the command to remove the modified Git files, my ./validate looks like this:

./validate.php

Component Version
LibreNMS 1.54-56-g7783776
DB Schema 2019_07_09_150217_update_users_widgets_settings (140)
PHP 7.2.19
MySQL 5.5.60-MariaDB
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 1.9.0
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[WARN] Your install is over 24 hours out of date, last update: Mon, 26 Aug 2019 14:42:27 +0000
[FIX]:
Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.
[FAIL] We have found some files that are owned by a different user than librenms, this will stop you updating automatically and / or rrd files being updated causing graphs to fail.
[FIX]:
sudo chown -R librenms:librenms /opt/librenms
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
Files:
/opt/librenms/.git/index
/opt/librenms/logs/.gitignore
/opt/librenms/rrd/.gitignore
/opt/librenms/storage/app/public/.gitignore
/opt/librenms/storage/app/.gitignore
/opt/librenms/storage/framework/testing/.gitignore
/opt/librenms/storage/logs/.gitignore

Again, the script I have will fix the user rights because even if I enter the suggested commands to give ownership towards my Librenms user it will revert itself.

Does anyone have any idea what I am doing wrong?

If it helps, my ./daily.sh runs fine if I enter the command:

./daily.sh
Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase OK
Updating Composer packages OK
Updated from 7783776 to 7811227 OK
Updating SQL-Schema OK
Updating submodules OK
Cleaning up DB OK
Fetching notifications OK
Caching PeeringDB data OK

Aside from this, LibreNMS runs perfectly fine. I hope someone can help.

github-remove -d will fix the .gitignore files.
Sometimes when something hasn’t been merged that day you get that message from daily.sh.

If you want to help that, go test and review some pull requests on github :stuck_out_tongue:

Might be a silly question, but have you followed the guidance at the top of the page…

su - librenms
git pull
./daily.sh
1 Like

Thanks for both your inputs! I will try those options.

Guess it’s fit here:

I’m getting the “Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.”

But not sure why. all seems up to date and good to go but keeps warning. any inputs ?

[email protected]:~$ ./daily.sh 
Updating to latest codebase                        OK
Updating Composer packages                         OK
Updating SQL-Schema                                OK
Updating submodules                                OK
Cleaning up DB                                     OK
Fetching notifications                             OK
Caching PeeringDB data                             OK
[email protected]:~$ ./validate.php 
====================================
Component | Version
--------- | -------
LibreNMS  | 1.59-531-gd9add2c
DB Schema | 2019_12_16_140000_create_customoids_table (152)
PHP       | 7.2.16-1+ubuntu16.04.1+deb.sury.org+1
MySQL     | 10.0.38-MariaDB-0ubuntu0.16.04.1
RRDTool   | 1.5.5
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.9.1
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[WARN]  Your install is over 24 hours out of date, last update: Sat, 04 Jan 2020 23:15:13 +0000
	[FIX]: 
	Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.
[email protected]:~$ cat /etc/cron.d/librenms 
# Using this cron file requires an additional user on your system, please see install docs.

#33  */6   * * *   librenms    /opt/librenms/discovery.php -h all >> /dev/null 2>&1
33   */6  * * *   librenms    /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1
*/5  *    * * *   librenms    /opt/librenms/discovery.php -h new >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/cronic /opt/librenms/poller-wrapper.py 4
15   0    * * *   librenms    /opt/librenms/daily.sh >> /dev/null 2>&1
*    *    * * *   librenms    /opt/librenms/alerts.php >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/poll-billing.php >> /dev/null 2>&1
01   *    * * *   librenms    /opt/librenms/billing-calculate.php >> /dev/null 2>&1
*/5  *    * * *   librenms    /opt/librenms/check-services.php >> /dev/null 2>&1
*/2  *    * * *   librenms    /opt/librenms/html/plugins/Weathermap/map-poller.php >> /dev/null 2>&1
[email protected]:~$ git pull
Already up-to-date.
[email protected]:~$ 
[email protected]:~$ ./daily.sh
Updating to latest codebase                        OK
Updating Composer packages                         OK
Updating SQL-Schema                                OK
Updating submodules                                OK
Cleaning up DB                                     OK
Fetching notifications                             OK
Caching PeeringDB data                             OK
[email protected]:~$ ./validate.php 
====================================
Component | Version
--------- | -------
LibreNMS  | 1.59-531-gd9add2c
DB Schema | 2019_12_16_140000_create_customoids_table (152)
PHP       | 7.2.16-1+ubuntu16.04.1+deb.sury.org+1
MySQL     | 10.0.38-MariaDB-0ubuntu0.16.04.1
RRDTool   | 1.5.5
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.9.1
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[WARN]  Your install is over 24 hours out of date, last update: Sat, 04 Jan 2020 23:15:13 +0000
	[FIX]: 
	Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.
[email protected]:~$

I think that logic for “Your install is over 24 hours out of date” checks the timezone of the system and compares it to the last git commit. If 1 day or more of difference it will throw that error. Since the last commit was 2 days ago i think it’s safe to ignore https://github.com/librenms/librenms/pulls?utf8=✓&q=is%3Amerged.

....
if ($commit_date->diff(new DateTime())->days > 0) {
                        $validator->warn(
                            "Your install is over 24 hours out of date
...
1 Like

Arh okay now I understand. :slight_smile: thanks for explaining to me.

See that there haven’t been done any updates for a coble of days:

just did an update after I so the 2 commits and correct the warning disappeared:

[email protected]:~$ date
Tue  7 Jan 15:04:47 CET 2020
[email protected]:~$ ./daily.sh 
Updating to latest codebase                        OK
Updating Composer packages                         OK
Updated from d9add2c to a65c9e8                    OK
Updating SQL-Schema                                OK
Updating submodules                                OK
Cleaning up DB                                     OK
Fetching notifications                             OK
Caching PeeringDB data                             OK
[email protected]:~$ ./validate.php 
====================================
Component | Version
--------- | -------
LibreNMS  | 1.59-534-ga65c9e8
DB Schema | 2019_12_16_140000_create_customoids_table (152)
PHP       | 7.2.26-1+ubuntu16.04.1+deb.sury.org+1
MySQL     | 10.0.38-MariaDB-0ubuntu0.16.04.1
RRDTool   | 1.5.5
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.9.1
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[email protected]:~$
1 Like