Daily.sh & validate.php issue - LibreNMS not updating itself

<%- if @topic_view.topic.tags.present? %>
<%= t 'js.tagging.tags' %>: <%- @topic_view.topic.tags.each do |t| %> <%= t %> <%- end %>
<% end %>

We are having issue were our install of LibreNMS doesn’t update by itself… looks like something may have been corrupted…

Output of daily.sh and validate.php gives the following:

bash-4.2$ ./daily.sh
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Updating to latest codebase FAIL
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Updating Composer packages

Tried to solve, but doesn’t have that much git skills :slight_smile: and needs help to get this resolved.

We do have a LibreNMS test machine (Clone of the production machine) that we can work with prior doing this on the production machine.

Both of them doing the exact same behavior as it’s a clone.

I can gives any kind of output requested :slight_smile:

Thanks for the help

Is that output just from daily.sh? noticed it finished on updating composer.

How long do you think it’s been like this? do you have any logs under logs/daily.log

Does git status show anything in your install directory ?

Are you running librenms on a network mounted disk or docker container ?

Do you happen to have a GIT_DIR environment variable set?

Hi Chas,

Yes the output is from daily.sh, I just stop capturing the text at “Updating Composer Packages”. The daily.sh fails following tasks: “Updating to latest codebase” and “Updating submodules”

Git status gives:

fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

LibreNMS is installed locally, the only things is that the original installation directory was moved. (Symlink from /opt/librenms to /app/librenms)

There’s no GIT_DIR variable in both of our installations.

I don’t know from when did the issue occurs, we did noticed this when we were looking for a feature released in april 2019 and it was missing. Our Librenms installation isn’t that old (November 2018)

Daily.log says this:

Updating to latest codebase
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Returned: 128
Updating Composer packages

[Composer\Downloader\TransportException]
The “https://getcomposer.org/versions” file could not be downloaded: failed to open stream: Connection timed out

self-update [-r|–rollback] [–clean-backups] [–no-progress] [–update-keys] [–stable] [–preview] [–snapshot] [–set-channel-only] [–] []

Updating submodules
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Returned: 128

Thanks for you help :slight_smile:

@Bob81 can you show us the output from git remote -v and git status?

Also, did you copy LibreNMS without the .git directory?

Hi Murrant,

The directory was moved from /opt/librenms to /app/librenms and we did a simlink to link them. But this is the strange thing… the .git directory doesn’t exist at all.

Will check on the backup when this folder dissapear, but if it was existing orignally it should have been moved with the mv.

Output of both command gives:

fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

Thanks :slight_smile:

If you can’t get the .git directory from backup, you can recreate it.

Doesn’t look to be available from the backup… Thoses machine aren’t keep that long in backup rotation.

How can we proceed to recreate the .git directory?

Thanks

git init
git remote add origin https://github.com/librenms/librenms.git
git fetch
git reset --hard origin/master
./validate.php

Librenms installation crashing right after the “git reset” command. Getting http/500. Even after running validate.php it asks to run some commands witch I did and still failing with http/500…

1st time validate.php output:

====================================

Component Version
LibreNMS 1.52-45-g0baeb07
DB Schema Not Connected (0)
PHP 7.2.17
MySQL ?
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2

====================================

[FAIL] Missing dependencies! [FIX] ./scripts/composer_wrapper.php install --no-dev
kylekatarnls/update-helper
symfony/polyfill-php72
symfony/polyfill-php73
symfony/polyfill-intl-idn
symfony/mime
symfony/event-dispatcher-contracts
symfony/service-contracts
symfony/polyfill-iconv
doctrine/lexer
egulias/email-validator
and 18 more…
[FAIL] Outdated dependencies [FIX] ./scripts/composer_wrapper.php install --no-dev
symfony/dotenv
amenadiel/jpgraph
doctrine/inflector
erusev/parsedown
symfony/polyfill-ctype
vlucas/phpdotenv
symfony/css-selector
symfony/polyfill-mbstring
symfony/var-dumper
symfony/routing
and 29 more…
[OK] Composer Version: 1.8.6
[FAIL] Missing dependencies!
[FIX]:
composer install --no-dev
Dependencies:
kylekatarnls/update-helper
symfony/polyfill-php72
symfony/polyfill-php73
symfony/polyfill-intl-idn
symfony/mime
symfony/event-dispatcher-contracts
symfony/service-contracts
symfony/polyfill-iconv
doctrine/lexer
egulias/email-validator
psr/simple-cache
psr/container
opis/closure
symfony/translation-contracts
ralouphie/getallheaders
and 13 more…

2nd time validate.php (After command were run):

====================================

Component Version
LibreNMS 1.52-45-g0baeb07
DB Schema No Schema (0)
PHP 7.2.17
MySQL 10.3.14-MariaDB
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 1.8.6
[OK] Dependencies up-to-date.
[OK] Database connection successful
[FAIL] Your database schema (271) is older than the latest (1000).
[FIX]:
Manually run ./daily.sh, and check for any errors.
[FAIL] The poller (s1186moniqcp.ssq.local) has not completed within the last 5 minutes, check the cron job.
[FAIL] Discovery has not completed in the last 24 hours.
[FIX]:
Check the cron job to make sure it is running and using discovery-wrapper.py
[FAIL] We have found some files that are owned by a different user than librenms, this will stop you updating automatically and / or rrd files being updated causing graphs to fail.
[FIX]:
sudo chown -R librenms:librenms /app/librenms
sudo setfacl -d -m g::rwx /app/librenms/rrd /app/librenms/logs /app/librenms/bootstrap/cache/ /app/librenms/storage/
sudo chmod -R ug=rwX /app/librenms/rrd /app/librenms/logs /app/librenms/bootstrap/cache/ /app/librenms/storage/
Files:
/app/librenms/disable_all_device.sql
/app/librenms/html/plugins/Weathermap/configs/Prod_Physical_FW.conf
/app/librenms/html/includes/graphs/generic_data.inc.php.orig
/app/librenms/includes/dbFacile.php_bck
/app/librenms/includes/polling/bck_netscaler-vsvr.inc.php

Run the daily.sh and it says this (It’s being stuck at Updating SQL-Schema):

Updating to latest codebase FAIL
There is no tracking information for the current branch.
Please specify which branch you want to merge with.
See git-pull(1) for details

git pull <remote> <branch>

If you wish to set tracking information for this branch you can do so with:

git branch --set-upstream-to=origin/<branch> master

Updating Composer packages OK
Updating SQL-Schema

I know about the security file warning, i will check this as all of thoses files are backups.

Thanks! :slight_smile:

So run git branch --set-upstream-to=origin/master master

You can run ./build-base.php to update the schema too

Although, I noticed it says no schema above, maybe check your database credentials are correct in your .env file.

.env file looks good. DB_HOST, DB_DATABASE, DB_USERNAME & DB_PASSWORD are there. Same value as in the config.php file.

I’m able to query and browse the database from the mysql client so the database is up and running.

Running the git branch command did resolve the daily.sh update to latest codebase.

Thanks!

I do see some strange thing in the librenms.log file that starts right after the “git reset” command:

production.ERROR: SQLSTATE[42S02]: Base table or view not found: 1146 Table ‘librenms.mpls_lsps’ doesn’t exist (SQL: select count(*) as aggregate from mpls_lsps) (View: /app/librenms/resources/views/layouts/librenmsv1.blade.php) (View: /app/librenms/resources/views/layouts/librenmsv1.blade.php) {“userId”:3,“exception”:"[object] (ErrorException(code: 0): SQLSTATE[42S02]:

No stress as we having those errors on a test machine, once we will have the good receipe we will apply it to the production machine :slight_smile:

Thanks

Any ideas? I’m running out of option on this one :frowning:

Thanks

Anyone having any ideas to help on this issue?

Thanks :slight_smile: