Updating SQL-Schema - hangs


#1

Hi everyone,

Having a major problem with the database. All of sudden ibdata1 started rapidly growing (1GB per hour) and despite stopping syslog and doing all possible purges I was not able to resolve this, so I’ve restored the Libre instance from a snapshot and now not able to update the SQL-Schema after adding extra 20GB to the filled disk:

[[email protected] librenms]# sudo -u librenms ./daily.sh
Could not connect to database, check logs/librenms.log.
Updating to latest codebase OK
Updating Composer packages OK
Updated from d303174 to 36b4d16 OK
Updating SQL-Schema

It just hangs forever and nothing happens.
Validate shows the following:

[[email protected] librenms]# sudo -u librenms ./validate.php

Component Version
LibreNMS 1.47-19-g36b4d16
DB Schema 277
PHP 7.0.31
MySQL 5.5.56-MariaDB
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 1.8.0
[OK] Dependencies up-to-date.
[OK] Database connection successful
[FAIL] Your database schema (277) is older than the latest (278).
[FIX]:
Manually run ./daily.sh, and check for any errors.
[WARN] Your local git contains modified files, this could prevent automatic updates.
[FIX]:
You can fix this with ./scripts/github-remove
Modified Files:
bootstrap/cache/.gitignore
storage/app/.gitignore
storage/app/public/.gitignore
storage/debugbar/.gitignore
storage/framework/cache/.gitignore
storage/framework/sessions/.gitignore
storage/framework/testing/.gitignore
storage/framework/views/.gitignore
storage/logs/.gitignore

WebUI is down and storage is still filling up at approx 1GB per hour:

[[email protected] librenms]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/cl_bchqmon01-root 70G 52G 19G 75% /
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 9.2M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda1 1014M 288M 727M 29% /boot
/dev/mapper/cl_bchqmon01-home 26G 34M 26G 1% /home
tmpfs 782M 0 782M 0% /run/user/996
tmpfs 782M 0 782M 0% /run/user/0

Could somebody please assist? I’ve tried to update the SQL schema manually from here: https://github.com/librenms/librenms/blob/master/sql-schema/278.sql but it just hangs the same way as when running daily.sh


#2

That could take awhile to run let it finish. Its from https://github.com/librenms/librenms/pull/9612


#3

I’m sure I left it to run last night before going home and nothing happened, but I will give another go and wait and see what happens. Thanks


#4

So unfortunately after 18 hours it hasn’t updated:

[[email protected] librenms]# sudo -u librenms ./validate.php

Component Version
LibreNMS 1.47-20-gef8d7ad
DB Schema 277
PHP 7.0.31
MySQL 5.5.56-MariaDB
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 1.8.0
[OK] Dependencies up-to-date.
[OK] Database connection successful
[FAIL] Your database schema (277) is older than the latest (279).
[FIX]:
Manually run ./daily.sh, and check for any errors.
[WARN] Your local git contains modified files, this could prevent automatic updates.
[FIX]:
You can fix this with ./scripts/github-remove
Modified Files:
bootstrap/cache/.gitignore
storage/app/.gitignore
storage/app/public/.gitignore
storage/debugbar/.gitignore
storage/framework/cache/.gitignore
storage/framework/sessions/.gitignore
storage/framework/testing/.gitignore
storage/framework/views/.gitignore
storage/logs/.gitignore

The storage has gone down by 10 gb overnight:

[[email protected] librenms]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/cl_bchqmon01-root 70G 63G 7.9G 89% /

Can somebody please help diagnose/fix the problem?
The system has been operating for a year with no storage/database issues and consistent amount of added nodes (~50 in total).


#5

You should run the fix validate is telling you to do