Librenms stopped working on Nov 11

I’ve been running librenms on a longtime-stable server Ubuntu 16.04.6 with few troubles, until just a few days ago.

On Nov 11 when I came in, the web pages were all coming in blank. A fairly routine manual “cd /opt/librenms; ./daily.sh” managed to fix that problem for me. Or so it seemed, because apparently it hasn’t been doing any polling.

Today again everything has come to a halt, but now worse. Now I was getting the same issue as was seen in Undefined index: name In PackageManifest.php line 122 - #2 by monsterciattolo

“Undefined index: name In PackageManifest.php line 122”

The quickfix there (manually editing the “vendor/laravel/framework/src/Illuminate/Foundation/PackageManifest.php” file as per “monsterciattolo” did allow ./daily.sh to run and the web interface is back.

But NONE of the polling is happening. And in fact looking at my https:///poller page shows that its last run was 2021-11-11 00:15:44

For now I’ve renamed the “daily.sh” script to “daily.sh-disabled” so that it won’t automatically run.

This is the output of ./validate.php as run by user librenms:

$ ./validate.php

Component Version
LibreNMS 21.11.0
DB Schema 2021_10_20_224207_increase_length_of_attrib_type_column (221)
PHP 7.3.17-1+ubuntu16.04.1+deb.sury.org+1
Python 3.5.2
MySQL 10.2.31-MariaDB-10.2.31+maria~xenial
RRDTool 1.5.5
SNMP NET-SNMP 5.7.3

====================================

[OK] Composer Version: 2.1.12
[OK] Dependencies up-to-date.
[OK] Database connection successful
[FAIL] Your database is out of date!
[FIX]:
./lnms migrate
[FAIL] Python3 module issue found: ‘’
[FIX]:
pip3 install -r /opt/librenms/requirements.txt
[WARN] Your local git contains modified files, this could prevent automatic updates.
[FIX]:
You can fix this with ./scripts/github-remove
Modified Files:
daily.sh

Is there a way to rollback to (say) last week’s version?

I had the same problem yesterday as well and after some searching and trying many different things, it turns out it was because the Python version (3.5) in 16.04 was deprecated and no longer supported.

Python 3.6 isn’t officially available on 16.04 unless you add an external repository. I went that route at first, but couldn’t get pip3 to work correctly.

Eventually, what I did was I updated from 16.04 to 18.04 and then re-ran ./daily.sh and ./validate.php.

You may need to do a ‘git checkout master’ to resync the branch. I had to do it on my end because when it failed, it was changed.

LibreNMS has 2 branch branch Development branch and Stable branch.

The stable version is monthly and released on or around the last Sunday of the month.

https://docs.librenms.org/General/Releases/

./daily.sh it updates the main LibreNMS files, and the database structure. You can disable the update process daily.sh Global WebUI Settings under System → Updates or using lnms:./lnms config: set update false

https://docs.librenms.org/General/Updating/

I am using LibreNMS on Ubuntu server 20.04 with Update Channel release and it is working fine.

No, Python 3.5 is supported for the python wrapper. What happened was:

  1. A bug that blocked python dependency update during daily.sh was fixed.
  2. PyMySQL version 1.0.0 dropped support of Python 3.5 but didn’t mark that in their setup. Causing breakage. (they yanked 1.0.0, but somehow it is still getting installed)
  3. Shortly after this was discovered, PyMySQL version 1.0.0 was specifically excluded in requirements.txt, so daily.sh would fix things back up installing a version supported by the local python version.

OP may have also corrupted some generated files, the correct fix is to regenerate them.

sudo su - librenms
rm -rf vendor
./scripts/composer_wrapper.php install --no-dev

What I would do I upgrade to 20.xx… at some point getting help for 16/18 will be hard to fine
I have two servers Ubuntu 18 running Librenms.
My upgrade path. ( i did the over a 3 weeks and spent about 27hrs)
I can now do the upgrade in about 8 hrs and 6 of that is transferring the data.

  1. Install Ubuntu 20 and Librenms on VM or server.
  2. Verified that this new installation has no problems.
    Ran daily.sh, validate.sh, apt update and installed rrdchache. Resolve all errors.
  3. Transfer the data base
    You could shutdown the database (old) here - i did not and had no problems
    mysqldump librenms -u root -p > librenms.sql the old system
    mysql -u root -p librenms < librenms.sql the new system
    FAQ - LibreNMS Docs
    Now you should have the device and accounts on the news system.
  4. Transfer the RRD files.
    New system:
    move librenms from /etc/cron.d
    Shut data base down with systemctl shut mariadb
    if you don’t shut the data base then rrd files will be created. When you move the rrd files from the old to new there will be unsolvable issues.
    delete the rrd directory “rmdir -r rrd”
    create the rrd directory --directory should be empty
    transfer the rrd directory from the old system to new. (my was about 6 hours)
    scp -r rrd [email protected]:~/rrd on the old system
    move librenms back to /etc/cron.d
    restart the database systemctl restart mariadb.

As I mention I have two systems on 18.
the OS upgrade from 18 to 20 on server 1 took about 9 hours.
On system 2 the upgrade failed and now I have to do fresh install.
Because I have 2 systems I was able to keep monitoring going.

Ah good to know! That makes total sense. :slight_smile:

[email protected]:~$ ./scripts/composer_wrapper.php install --no-dev
> LibreNMS\ComposerHelper::preInstall
Installing dependencies from lock file
Verifying lock file contents can be installed on current platform.
Nothing to install, update or remove
Generating optimized autoload files
> Illuminate\Foundation\ComposerScripts::postAutoloadDump
> @php artisan package:discover --ansi
Discovered Package: fruitcake/laravel-cors
Discovered Package: genealabs/laravel-caffeine
Discovered Package: laravel-notification-channels/webpush
Discovered Package: laravel/tinker
Discovered Package: laravel/ui
Discovered Package: librenms/laravel-vue-i18n-generator
Discovered Package: mews/purifier
Discovered Package: nesbot/carbon
Discovered Package: php-flasher/flasher-laravel
Discovered Package: tightenco/ziggy
Discovered Package: wpb/string-blade-compiler
Package manifest generated successfully.
67 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
> LibreNMS\ComposerHelper::postInstall
> Illuminate\Foundation\ComposerScripts::postInstall
> @php artisan vue-i18n:generate --multi-locales --format=umd
> @php artisan view:cache
Compiled views cleared!
Blade templates cached successfully!
> @php artisan optimize
Configuration cache cleared!
Configuration cached successfully!
Route cache cleared!
Routes cached successfully!
Files cached successfully!
> @php artisan config:clear
Configuration cache cleared!
> scripts/dynamic_check_requirements.py || pip3 install --user -r requirements.txt || :
Traceback (most recent call last):
  File "scripts/dynamic_check_requirements.py", line 12, in <module>
    import pkg_resources
  File "/opt/librenms/.local/lib/python3.5/site-packages/pkg_resources/__init__.py", line 117
    f"{v} is an invalid version and will not be supported in "
                                                             ^
SyntaxError: invalid syntax
Traceback (most recent call last):
  File "/usr/bin/pip3", line 11, in <module>
    sys.exit(main())
  File "/opt/librenms/.local/lib/python3.5/site-packages/pip/__init__.py", line 11, in main
    from pip._internal.utils.entrypoints import _wrapper
  File "/opt/librenms/.local/lib/python3.5/site-packages/pip/_internal/utils/entrypoints.py", line 4, in <module>
    from pip._internal.cli.main import main
  File "/opt/librenms/.local/lib/python3.5/site-packages/pip/_internal/cli/main.py", line 57
    sys.stderr.write(f"ERROR: {exc}")
                                   ^
SyntaxError: invalid syntax

Did not help for me.

F string formating is python 3.6 feature and not available below versions.

1 Like

Yes, I already solved the problem.