Daily.sh & validate.php issue - LibreNMS not updating itself

Bob81 · 10 June 2019 19:34

We are having issue were our install of LibreNMS doesn’t update by itself… looks like something may have been corrupted…

Output of daily.sh and validate.php gives the following:

bash-4.2$ ./daily.sh
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Updating to latest codebase FAIL
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Updating Composer packages

Tried to solve, but doesn’t have that much git skills and needs help to get this resolved.

We do have a LibreNMS test machine (Clone of the production machine) that we can work with prior doing this on the production machine.

Both of them doing the exact same behavior as it’s a clone.

I can gives any kind of output requested

Thanks for the help

Chas · 10 June 2019 22:13

Is that output just from daily.sh? noticed it finished on updating composer.

How long do you think it’s been like this? do you have any logs under logs/daily.log

Does git status show anything in your install directory ?

Are you running librenms on a network mounted disk or docker container ?

Do you happen to have a GIT_DIR environment variable set?

Bob81 · 11 June 2019 12:34

Hi Chas,

Yes the output is from daily.sh, I just stop capturing the text at “Updating Composer Packages”. The daily.sh fails following tasks: “Updating to latest codebase” and “Updating submodules”

Git status gives:

fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

LibreNMS is installed locally, the only things is that the original installation directory was moved. (Symlink from /opt/librenms to /app/librenms)

There’s no GIT_DIR variable in both of our installations.

I don’t know from when did the issue occurs, we did noticed this when we were looking for a feature released in april 2019 and it was missing. Our Librenms installation isn’t that old (November 2018)

Daily.log says this:

Updating to latest codebase
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Returned: 128
Updating Composer packages

[Composer\Downloader\TransportException]
The “https://getcomposer.org/versions” file could not be downloaded: failed to open stream: Connection timed out

self-update [-r|–rollback] [–clean-backups] [–no-progress] [–update-keys] [–stable] [–preview] [–snapshot] [–set-channel-only] [–] []

Updating submodules
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Returned: 128

Thanks for you help

murrant · 11 June 2019 12:46

@Bob81 can you show us the output from git remote -v and git status?

Also, did you copy LibreNMS without the .git directory?

Bob81 · 11 June 2019 14:49

Hi Murrant,

The directory was moved from /opt/librenms to /app/librenms and we did a simlink to link them. But this is the strange thing… the .git directory doesn’t exist at all.

Will check on the backup when this folder dissapear, but if it was existing orignally it should have been moved with the mv.

Output of both command gives:

fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

Thanks

murrant · 11 June 2019 16:50

If you can’t get the .git directory from backup, you can recreate it.

Bob81 · 11 June 2019 19:55

Doesn’t look to be available from the backup… Thoses machine aren’t keep that long in backup rotation.

How can we proceed to recreate the .git directory?

Thanks

murrant · 11 June 2019 20:29

git init
git remote add origin https://github.com/librenms/librenms.git
git fetch
git reset --hard origin/master
./validate.php

Bob81 · 12 June 2019 13:15

Librenms installation crashing right after the “git reset” command. Getting http/500. Even after running validate.php it asks to run some commands witch I did and still failing with http/500…

1st time validate.php output:

====================================

Component Version

LibreNMS 1.52-45-g0baeb07

DB Schema Not Connected (0)

PHP 7.2.17

MySQL ?

RRDTool 1.4.8

SNMP NET-SNMP 5.7.2

====================================

[FAIL] Missing dependencies! [FIX] ./scripts/composer_wrapper.php install --no-dev
kylekatarnls/update-helper
symfony/polyfill-php72
symfony/polyfill-php73
symfony/polyfill-intl-idn
symfony/mime
symfony/event-dispatcher-contracts
symfony/service-contracts
symfony/polyfill-iconv
doctrine/lexer
egulias/email-validator
and 18 more…
[FAIL] Outdated dependencies [FIX] ./scripts/composer_wrapper.php install --no-dev
symfony/dotenv
amenadiel/jpgraph
doctrine/inflector
erusev/parsedown
symfony/polyfill-ctype
vlucas/phpdotenv
symfony/css-selector
symfony/polyfill-mbstring
symfony/var-dumper
symfony/routing
and 29 more…
[OK] Composer Version: 1.8.6
[FAIL] Missing dependencies!
[FIX]:
composer install --no-dev
Dependencies:
kylekatarnls/update-helper
symfony/polyfill-php72
symfony/polyfill-php73
symfony/polyfill-intl-idn
symfony/mime
symfony/event-dispatcher-contracts
symfony/service-contracts
symfony/polyfill-iconv
doctrine/lexer
egulias/email-validator
psr/simple-cache
psr/container
opis/closure
symfony/translation-contracts
ralouphie/getallheaders
and 13 more…

2nd time validate.php (After command were run):

====================================

Component Version

LibreNMS 1.52-45-g0baeb07

DB Schema No Schema (0)

PHP 7.2.17

MySQL 10.3.14-MariaDB

RRDTool 1.4.8

SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 1.8.6
[OK] Dependencies up-to-date.
[OK] Database connection successful
[FAIL] Your database schema (271) is older than the latest (1000).
[FIX]:
Manually run ./daily.sh, and check for any errors.
[FAIL] The poller (s1186moniqcp.ssq.local) has not completed within the last 5 minutes, check the cron job.
[FAIL] Discovery has not completed in the last 24 hours.
[FIX]:
Check the cron job to make sure it is running and using discovery-wrapper.py
[FAIL] We have found some files that are owned by a different user than librenms, this will stop you updating automatically and / or rrd files being updated causing graphs to fail.
[FIX]:
sudo chown -R librenms:librenms /app/librenms
sudo setfacl -d -m g::rwx /app/librenms/rrd /app/librenms/logs /app/librenms/bootstrap/cache/ /app/librenms/storage/
sudo chmod -R ug=rwX /app/librenms/rrd /app/librenms/logs /app/librenms/bootstrap/cache/ /app/librenms/storage/
Files:
/app/librenms/disable_all_device.sql
/app/librenms/html/plugins/Weathermap/configs/Prod_Physical_FW.conf
/app/librenms/html/includes/graphs/generic_data.inc.php.orig
/app/librenms/includes/dbFacile.php_bck
/app/librenms/includes/polling/bck_netscaler-vsvr.inc.php

Run the daily.sh and it says this (It’s being stuck at Updating SQL-Schema):

Updating to latest codebase FAIL
There is no tracking information for the current branch.
Please specify which branch you want to merge with.
See git-pull(1) for details
git pull <remote> <branch>
If you wish to set tracking information for this branch you can do so with:
git branch --set-upstream-to=origin/<branch> master
Updating Composer packages OK
Updating SQL-Schema

I know about the security file warning, i will check this as all of thoses files are backups.

Thanks!

murrant · 12 June 2019 14:06

So run git branch --set-upstream-to=origin/master master

murrant · 12 June 2019 14:07

You can run ./build-base.php to update the schema too

murrant · 12 June 2019 14:08

Although, I noticed it says no schema above, maybe check your database credentials are correct in your .env file.

Bob81 · 12 June 2019 14:28

.env file looks good. DB_HOST, DB_DATABASE, DB_USERNAME & DB_PASSWORD are there. Same value as in the config.php file.

I’m able to query and browse the database from the mysql client so the database is up and running.

Running the git branch command did resolve the daily.sh update to latest codebase.

Thanks!

Bob81 · 13 June 2019 15:07

I do see some strange thing in the librenms.log file that starts right after the “git reset” command:

production.ERROR: SQLSTATE[42S02]: Base table or view not found: 1146 Table ‘librenms.mpls_lsps’ doesn’t exist (SQL: select count(*) as aggregate from mpls_lsps) (View: /app/librenms/resources/views/layouts/librenmsv1.blade.php) (View: /app/librenms/resources/views/layouts/librenmsv1.blade.php) {“userId”:3,“exception”:"[object] (ErrorException(code: 0): SQLSTATE[42S02]:

No stress as we having those errors on a test machine, once we will have the good receipe we will apply it to the production machine

Thanks

Bob81 · 26 June 2019 13:42

Any ideas? I’m running out of option on this one

Thanks

Bob81 · 24 July 2019 12:35

Anyone having any ideas to help on this issue?

Thanks

Bob81 · 6 November 2019 15:20

Still having the same issue, still needs help on getting this resolved

Ricardo_Viteri · 6 November 2019 21:21

Same issue here…

root@librenms:/opt/librenms# ./daily.sh
Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase FAIL
error: object file .git/objects/22/3a5b15dc238bba6858357582e97dd6c5275dd6 is empty
error: object file .git/objects/22/3a5b15dc238bba6858357582e97dd6c5275dd6 is empty
fatal: loose object 223a5b15dc238bba6858357582e97dd6c5275dd6 (stored in .git/objects/22/3a5b15dc238bba6858357582e97dd6c5275dd6) is corrupt
fatal: The remote end hung up unexpectedly
Updating Composer packages OK
Updating SQL-Schema OK
Updating submodules OK
Cleaning up DB OK
Fetching notifications OK
Caching PeeringDB data OK

murrant · 7 November 2019 02:10

@Bob81 your database structure is not correct, run ./validate.php

murrant · 7 November 2019 02:14

Your issue is totally different. Your .git directory is corrupted.

rm -rf .git
git init
git remote add origin https://github.com/librenms/librenms.git
git fetch
git reset origin/master --mixed

Component	Version
LibreNMS	1.52-45-g0baeb07
DB Schema	Not Connected (0)
PHP	7.2.17
MySQL	?
RRDTool	1.4.8
SNMP	NET-SNMP 5.7.2
====================================