Update broke "old" LibreNMS

Hi everyone,

I’ve let update froze on my LibreNMS for some time now, here is the details:

[email protected]:/opt/librenms# ./validate.php 
====================================
Component | Version
--------- | -------
LibreNMS  | 1.52-70-gf3ba8947f
DB Schema | 2019_05_30_225937_device_groups_rewrite (135)
PHP       | 7.3.14-1+0~20200123.51+debian10~1.gbpcf42df
MySQL     | 10.3.21-MariaDB-1:10.3.21+maria~buster
RRDTool   | 1.7.1
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.9.2
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[WARN]  Your install is over 24 hours out of date, last update: Sun, 23 Jun 2019 05:29:12 +0000
        [FIX]: 
        Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.
[email protected]:/opt/librenms# ./daily.sh
Re-running /opt/librenms/daily.sh as librenms user
Updating SQL-Schema                                OK
Cleaning up DB                                     OK
[email protected]:/opt/librenms#

As you see, daily.sh don’t do the update anymore.
I find a trick with git fetch to get daily.sh to work again, but it broke librenms.

Everything works expect device page. And I don’t get any error trace on the librenms.log, just 500 error in the nginx access.

Also, it made re appear some old, deleted devices. I don’t know why.

What is the proper way to upgrade this?

Thanks and regards,

how about this

su - librenms
git pull
./daily.sh

Hi,

Same result, it basically works, but after I have my devices page that are blank with an error.
And 3 old devices re-appear.

Thanks,

Please run ./validate.php now, and attach the display.

Here:

./validate.php
====================================
Component | Version
--------- | -------
LibreNMS | 1.59-71-g7950893cd
DB Schema | 2020_01_09_1300_migrate_devices_attribs_table (156)
PHP | 7.3.14-1+0~20200123.51+debian10~1.gbpcf42df
MySQL | 10.3.22-MariaDB-1:10.3.22+maria~buster
RRDTool | 1.7.1
SNMP | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.9.2
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[WARN]  Your local git contains modified files, this could prevent automatic updates.
        [FIX]: 
        You can fix this with ./scripts/github-remove
        Modified Files:
         bootstrap/cache/.gitignore
         logs/.gitignore
         rrd/.gitignore
         storage/app/.gitignore
         storage/app/public/.gitignore
         storage/debugbar/.gitignore
         storage/framework/cache/.gitignore
         storage/framework/cache/data/.gitignore
         storage/framework/sessions/.gitignore
         storage/framework/testing/.gitignore
         storage/framework/views/.gitignore
         storage/logs/.gitignore

Device page works fine:

But I have this error on the specific device page:

And I have no error in librenms.log, only the polling and discovery details that works fine.

Edit 1: Also, 3 old deleted devices came back:

Edit 2:

It broke the poller also it seems:

[email protected]:/opt/librenms# /opt/librenms/poller-wrapper.py 6
INFO: starting the poller at 2020-01-31 10:24:14 with 6 threads, slowest devices first
INFO: worker Thread-5 finished device 83 in 0 seconds
INFO: worker Thread-2 finished device 39 in 0 seconds
INFO: worker Thread-1 finished device 79 in 0 seconds
INFO: worker Thread-4 finished device 78 in 0 seconds
INFO: worker Thread-3 finished device 77 in 0 seconds
INFO: worker Thread-6 finished device 80 in 0 seconds
INFO: worker Thread-5 finished device 81 in 0 seconds
INFO: worker Thread-4 finished device 71 in 0 seconds
INFO: worker Thread-2 finished device 67 in 0 seconds
INFO: worker Thread-6 finished device 51 in 0 seconds
INFO: worker Thread-1 finished device 54 in 0 seconds
INFO: worker Thread-3 finished device 68 in 0 seconds
INFO: worker Thread-5 finished device 60 in 0 seconds
INFO: worker Thread-1 finished device 70 in 0 seconds
INFO: worker Thread-4 finished device 75 in 0 seconds
INFO: worker Thread-3 finished device 50 in 0 seconds
INFO: worker Thread-2 finished device 52 in 0 seconds
INFO: worker Thread-6 finished device 49 in 0 seconds
INFO: worker Thread-5 finished device 66 in 0 seconds
INFO: worker Thread-2 finished device 74 in 0 seconds
INFO: worker Thread-1 finished device 72 in 0 seconds
INFO: worker Thread-4 finished device 57 in 0 seconds
INFO: worker Thread-3 finished device 84 in 0 seconds
INFO: worker Thread-6 finished device 73 in 0 seconds
INFO: poller-wrapper polled 24 devices in 2 seconds with 6 workers

Bonjour,

Any details on the device that does not display correctly ? (type, anything specific) ?

Could you load this broken device page, and grab the error in ./logs/librenms.log to get a hint on what’s going wrong (you may need to do it between to run of the poller, to avoid logs mixed between poller and GUI).

Could you try the following series of commands :
./daily.sh

  • (should do it’s stuff normally)

git fetch

  • (should return empty)

git diff origin master

  • (should return empty as well)

Bonne journée sous la grisaille genevoise …

Bonjour :slight_smile:

The problem is that I don’t have any error in the librenms.log, I just show the empty results from the poller.

No error when the page loads with the red panel ?

What about those commands below ?

./daily.sh

  • (should do it’s stuff normally)

git fetch

  • (should return empty)

git diff origin master

  • (should return empty as well)

Sorry for the delay, much to do but not on my monitoring :slight_smile:

[email protected]:/opt/librenms# ./daily.sh
Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase OK
Updating Composer packages OK
Updated from f8ce4df0f to 81454ee58 OK
Updating SQL-Schema OK
Updating submodules OK
Cleaning up DB OK
Fetching notifications OK
Caching PeeringDB data OK
[email protected]:/opt/librenms# git fetch
[email protected]:/opt/librenms# git diff origin master
[email protected]:/opt/librenms#

I see that you are running your commands with user root. This is not at all a good idea for permissions and rights. Please switch to “librenms” user, and check all (validate.php, daily.sh) and correct any error you’d see.
Concerning a device which would re-appear, I see no way it could happen except if the database was involved, one way or the other. Any idea of something “weird” on mysql side ?

The result is the same.

The problem appeared right after the update of the mysql schema by the daily script.
I think it just killed the database, I don’t see any problem on the “files” side or web server one.