Mysqld High cpu , Polling not completing

Help
1

Hi,

At some point in the last 12 hours our LibreNMS install stopped be able to poll all devices in time, gappy graphs etc.
A quick glance at top, and mysqld is being a cpu hog, and i’m only occasionally not able to connect to the DB. - Previously been running for about a month, and a reboot hasn’t helped anything

====================================

Component Version
LibreNMS 1.52-50-gcfc51d51f
DB Schema 2019_05_30_225937_device_groups_rewrite (135)
PHP 7.2.19-0ubuntu0.18.04.1
MySQL 10.1.40-MariaDB-0ubuntu0.18.04.1
RRDTool 1.7.0
SNMP NET-SNMP 5.7.3

====================================

[OK] Composer Version: 1.8.6
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[WARN] Some devices have not been polled in the last 5 minutes. You may have performance issues.
[FIX]:
Check your poll log and see: http://docs.librenms.org/Support/Performance/
Devices:
Devices removed
and 317 more…

Thanks in advance for your help!

2

https://pastebin.com/cs8ZMPZC

Tail of librenms.log above

3

Run updates again it was the device groups update that didnt work. the update removes it.

1 Like
4

Ok cool, I ran ./daily.sh and got the below -

Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase                        OK
Updating Composer packages                         OK
Updated from cfc51d51f to 2da6c8104                OK
Updating SQL-Schema                                OK
Updating submodules                                OK
Cleaning up DB                                     OK
Fetching notifications                             OK
Caching PeeringDB data                             OK

Now validate.php brings up the below…

====================================

Component Version
LibreNMS 1.52-52-g2da6c8104
DB Schema 2019_05_30_225937_device_groups_rewrite (135)
PHP 7.2.19-0ubuntu0.18.04.1
MySQL 10.1.40-MariaDB-0ubuntu0.18.04.1
RRDTool 1.7.0
SNMP NET-SNMP 5.7.3
====================================

[OK] Composer Version: 1.8.6
[OK] Dependencies up-to-date.
[OK] Database connection successful
[WARN] Your database schema has extra migrations (2019_05_30_225937_device_groups_rewrite). If you just switched to the stable release from the daily release, your database is in between releases and this will be resolved with the next release.
[FAIL] Database: incorrect column (device_groups/desc)
[FAIL] Database: missing column (device_groups/params)
[FAIL] Database: extra column (device_groups/type)
[FAIL] Database: extra column (device_groups/rules)
[FAIL] We have detected that your database schema may be wrong, please report the following to us on Discord (LibreNMS) or the community site (Report database schema issues here - LibreNMS Community):
[FIX]:
Run the following SQL statements to fix.
SQL Statements:
ALTER TABLE device_groups CHANGE desc desc varchar(255) NOT NULL DEFAULT ‘’ ;
ALTER TABLE device_groups ADD params text NULL AFTER pattern;
ALTER TABLE device_groups DROP type;
ALTER TABLE device_groups DROP rules;
[WARN] Some devices have not been polled in the last 5 minutes. You may have performance issues.
[FIX]:
Check your poll log and see: Performance - LibreNMS Docs
Devices:

Anything else other than the suggested SQL statements I should make or be concerned by? The Libre GUI is starting to look more normal again, although polling hasn’t caught up with itself

5

Switched myself over to the monthly update for now - All looks to be back to normal, thanks for the help!

Chris