Mysqld and rrdcached exhausting RAM after upgrade to php7.2

Hi! After Upgrade to php 7.2 performance is dropping, and i notice now mysqld & rrdcached exhausting my ram :frowning:

Server:

  • Ubuntu 16.04
  • 64 vCpu
  • 16 GB Ram

top - 12:37:40 up 1 day, 3:52, 2 users, load average: 1075.99, 1061.78, 1015.85
Tasks: 4967 total, 2 running, 4348 sleeping, 0 stopped, 302 zombie
%Cpu(s): 1.2 us, 0.7 sy, 0.0 ni, 0.0 id, 98.0 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 16413132 total, 342820 free, 13983068 used, 2087244 buff/cache
KiB Swap: 16777212 total, 6542352 free, 10234860 used. 694884 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
11181 mysql     20   0 37.328g 2.432g  10032 S  50.3 15.5  84:26.02 mysqld
 2656 librenms  20   0 36.341g 814936   2464 S  17.4  5.0 522:53.39 rrdcached

================./daily.sh===================
root@librenms:/opt/librenms# ./daily.sh
Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase                        OK
Updating Composer packages                         OK
Updated from cf17016 to 25954cc                    OK
Updating SQL-Schema                                OK
Updating submodules                                OK
Cleaning up DB                                     OK
Fetching notifications                             OK
Caching PeeringDB data                             OK


================./validate.php===================
root@librenms:/opt/librenms# ./validate.php
====================================
Component | Version
--------- | -------
LibreNMS  | 1.48.1-32-g25954cc
DB Schema | 2019_01_16_195644_add_vrf_id_and_bgpLocalAs (131)
PHP       | 7.2.15-1+ubuntu16.04.1+deb.sury.org+1
MySQL     | 10.3.12-MariaDB-1:10.3.12+maria~xenial
RRDTool   | 1.5.5
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.8.4
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[FAIL]  Discovery has not completed in the last 24 hours.
        [FIX]:
        Check the cron job to make sure it is running and using discovery-wrapper.py
```

[mysqld]
max_connections         = 500
wait_timeout            = 20000
#interactive_timeut     = 21600
query_cache_size        = 0
query_cache_type        = 0
tmp_table_size          = 32M
max_heap_table_size     = 32M
performance_schema      = ON
join_buffer_size        = 4096
innodb_buffer_pool_size = 4G
innodb_log_file_size    = 2M

How many devices/ports/sensors are you monitoring?

And how was it before upgrade?

Hi!
Number of device/sensor is still same before and after upgrade

image

Sorry, I mean how it was about memory usage

memory is not issue after the update.
after update mysqld & rrdcached exhausting memory and the swap and the system crashed. reboot is only the solution :frowning:

nevermind. I give up and roll back to last snapshoot.

Its ok.

Anyways, I think something went wrong in your php update to 7.2.

What OS?

ubuntu 16.04