Performance Issue After PHP Update

Hi! I’am struggling ater update Php from 7.0 to 7.2. some of my graph is intermittent and warning max_connection always show up. I have re-run the mysql tunner and follow the instruction.

here the source that i’m following for upgrade PHP:

==========my.cnf configuration===========
[mysqld]
max_connections         = 250
wait_timeout            = 20000
query_cache_size        = 0
query_cache_type        = 0
performance_schema      = ON
join_buffer_size        = 2048
innodb_buffer_pool_size = 8G
innodb_buffer_pool_size = 16M
innodb_flush_log_at_trx_commit = 0
=============================================

> ==========validate.sh===============
> Component | Version
> --------- | -------
> LibreNMS  | 1.48.1-30-g7e18a07
> DB Schema | 2019_01_16_195644_add_vrf_id_and_bgpLocalAs (131)
> PHP       | 7.2.15-1+ubuntu16.04.1+deb.sury.org+1
> MySQL     | 10.3.12-MariaDB-1:10.3.12+maria~xenial
> RRDTool   | 1.5.5
> SNMP      | NET-SNMP 5.7.3
> ====================================
> 
> [OK]    Composer Version: 1.8.4
> [OK]    Dependencies up-to-date.
> [OK]    Database connection successful
> [OK]    Database schema correct
> [FAIL]  The poller (librenms) has not completed within the last 5 minutes, check the cron job.
> [WARN]  Some devices have not been polled in the last 5 minutes. You may have performance issues.
>         [FIX]:
>         Check your poll log and see: http://docs.librenms.org/Support/Performance/
>         Devices:
>          localhost
>          some IP address
>           and 697 more...
> [FAIL]  Some devices have not completed their polling run in 5 minutes, this will create gaps in data.
>         [FIX]:
>         Check your poll log and see: http://docs.librenms.org/Support/Performance/
>         Devices:
>          ===some IP Address===
> [FAIL]  Discovery has not completed in the last 24 hours.
>         [FIX]:
>         Check the cron job to make sure it is running and using discovery-wrapper.py
> [FAIL]  Some folders have incorrect file permissions, this may cause issues.
>         [FIX]:
>         sudo chown -R librenms:librenms /opt/librenms
>         sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
>         sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
>         Files:
>          /opt/librenms/rrd/10.80.16.133/port-id.rrd
> -------------------------------------------------------------------------------------------------------------------

============daily.sh==========
Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase                        OK
Updating Composer packages                         OK
Updating SQL-Schema                                OK
Updating submodules                                OK
Cleaning up DB                                     OK
Fetching notifications                             OK
Caching PeeringDB data                             OK
=============================

your help is appreciated

I suggest you to update your mysql max connections to at least 500.

Said that. If that happens since you updated your PHP to 7.2, I would start doing a manual ./poller.php -d -h all and check where is hanging out the poller.

Hi!
Try to update max_connection value to 500. and after restart mysql daemon I check the librenms.log the poller stop working :’(

but I notice new log:

[2019-02-11 22:46:46] production.ERROR: SQLSTATE[42000]: Syntax error or access violation: 1066 Not unique table/alias: 'devices' (SQL: SELECT DISTINCT(devices.device_id) FROM devices,locations,devices WHERE (( locations.id = devices.location_id && devices.device_id = devices.device_id ) && ( devices.device_id = devices.device_id )) && devices.device_id=455 AND (locations.location REGEXP "jatinegara" || locations.location REGEXP "ancol" || locations.location REGEXP "pluit" || locations.location REGEXP "buaran" || devices.sysName REGEXP "bks" || locations.location REGEXP "bekasi" || locations.location REGEXP "sunter" || locations.location REGEXP "karawang" ) LIMIT 1) (SQL: SELECT DISTINCT(devices.device_id) FROM devices,locations,devices WHERE (( locations.id = devices.location_id && devices.device_id = devices.device_id ) && ( devices.device_id = devices.device_id )) && devices.device_id=455 AND (locations.location REGEXP "jatinegara" || locations.location REGEXP "ancol" || locations.location REGEXP "pluit" || locations.location REGEXP "buaran" || devices.sysName REGEXP "bks" || locations.location REGEXP "bekasi" || locations.location REGEXP "sunter" || locations.location REGEXP "karawang" ) LIMIT 1)

please advice

Looks like you have an invalid device group rule. Not related to the update at all.

Try group all items in the same table together.

Hi murrant!

I’ve removed the device group rules related the previous issue. Now, the error log not appear :smiley:
But I still struggling with graph issue :frowning:
libre

================./daily.sh===================
root@librenms:/opt/librenms# ./daily.sh
Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase                        OK
Updating Composer packages                         OK
Updated from cf17016 to 25954cc                    OK
Updating SQL-Schema                                OK
Updating submodules                                OK
Cleaning up DB                                     OK
Fetching notifications                             OK
Caching PeeringDB data                             OK


================./validate.php===================
root@librenms:/opt/librenms# ./validate.php
====================================
Component | Version
--------- | -------
LibreNMS  | 1.48.1-32-g25954cc
DB Schema | 2019_01_16_195644_add_vrf_id_and_bgpLocalAs (131)
PHP       | 7.2.15-1+ubuntu16.04.1+deb.sury.org+1
MySQL     | 10.3.12-MariaDB-1:10.3.12+maria~xenial
RRDTool   | 1.5.5
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.8.4
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[FAIL]  Discovery has not completed in the last 24 hours.
        [FIX]:
        Check the cron job to make sure it is running and using discovery-wrapper.py

================my.cnf===================
root@librenms:/opt/librenms# cat /etc/mysql/my.cnf
[mysqld]
max_connections         = 500
wait_timeout            = 20000
#interactive_timeut     = 21600
query_cache_size        = 0
query_cache_type        = 0
performance_schema      = ON
join_buffer_size        = 2048
innodb_buffer_pool_size = 8G
innodb_buffer_pool_size = 16M
#general_log_file        = /var/log/mysql/mysql.log
#general_log             = 1
innodb_flush_log_at_trx_commit = 0

please advice :pray:

Hi, seems the issue has been resolved.

The last command i’ve enter:

#systemctl restart rrdcached.service
#apt-get install php7.2-ldap
#apt-get install php7.2-snmp
#systemctl restart apache2

Perfect! Glad you solved it.