Mysqld High cpu , Polling not completing

Hi,

At some point in the last 12 hours our LibreNMS install stopped be able to poll all devices in time, gappy graphs etc.
A quick glance at top, and mysqld is being a cpu hog, and i’m only occasionally not able to connect to the DB. - Previously been running for about a month, and a reboot hasn’t helped anything

====================================

Component Version
LibreNMS 1.52-50-gcfc51d51f
DB Schema 2019_05_30_225937_device_groups_rewrite (135)
PHP 7.2.19-0ubuntu0.18.04.1
MySQL 10.1.40-MariaDB-0ubuntu0.18.04.1
RRDTool 1.7.0
SNMP NET-SNMP 5.7.3

====================================

[OK] Composer Version: 1.8.6
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[WARN] Some devices have not been polled in the last 5 minutes. You may have performance issues.
[FIX]:
Check your poll log and see: http://docs.librenms.org/Support/Performance/
Devices:
Devices removed
and 317 more…

Thanks in advance for your help!

https://pastebin.com/cs8ZMPZC

Tail of librenms.log above

Run updates again it was the device groups update that didnt work. the update removes it.

1 Like

Ok cool, I ran ./daily.sh and got the below -

Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase                        OK
Updating Composer packages                         OK
Updated from cfc51d51f to 2da6c8104                OK
Updating SQL-Schema                                OK
Updating submodules                                OK
Cleaning up DB                                     OK
Fetching notifications                             OK
Caching PeeringDB data                             OK

Now validate.php brings up the below…

====================================

Component Version
LibreNMS 1.52-52-g2da6c8104
DB Schema 2019_05_30_225937_device_groups_rewrite (135)
PHP 7.2.19-0ubuntu0.18.04.1
MySQL 10.1.40-MariaDB-0ubuntu0.18.04.1
RRDTool 1.7.0
SNMP NET-SNMP 5.7.3

====================================

[OK] Composer Version: 1.8.6
[OK] Dependencies up-to-date.
[OK] Database connection successful
[WARN] Your database schema has extra migrations (2019_05_30_225937_device_groups_rewrite). If you just switched to the stable release from the daily release, your database is in between releases and this will be resolved with the next release.
[FAIL] Database: incorrect column (device_groups/desc)
[FAIL] Database: missing column (device_groups/params)
[FAIL] Database: extra column (device_groups/type)
[FAIL] Database: extra column (device_groups/rules)
[FAIL] We have detected that your database schema may be wrong, please report the following to us on Discord (https://t.libren.ms/discord) or the community site (https://t.libren.ms/5gscd):
[FIX]:
Run the following SQL statements to fix.
SQL Statements:
ALTER TABLE device_groups CHANGE desc desc varchar(255) NOT NULL DEFAULT ‘’ ;
ALTER TABLE device_groups ADD params text NULL AFTER pattern;
ALTER TABLE device_groups DROP type;
ALTER TABLE device_groups DROP rules;
[WARN] Some devices have not been polled in the last 5 minutes. You may have performance issues.
[FIX]:
Check your poll log and see: http://docs.librenms.org/Support/Performance/
Devices:

Anything else other than the suggested SQL statements I should make or be concerned by? The Libre GUI is starting to look more normal again, although polling hasn’t caught up with itself

Switched myself over to the monthly update for now - All looks to be back to normal, thanks for the help!

Chris