Performance Issue After PHP Update


#1

Hi! I’am struggling ater update Php from 7.0 to 7.2. some of my graph is intermittent and warning max_connection always show up. I have re-run the mysql tunner and follow the instruction.

here the source that i’m following for upgrade PHP:

==========my.cnf configuration===========
[mysqld]
max_connections         = 250
wait_timeout            = 20000
query_cache_size        = 0
query_cache_type        = 0
performance_schema      = ON
join_buffer_size        = 2048
innodb_buffer_pool_size = 8G
innodb_buffer_pool_size = 16M
innodb_flush_log_at_trx_commit = 0
=============================================

> ==========validate.sh===============
> Component | Version
> --------- | -------
> LibreNMS  | 1.48.1-30-g7e18a07
> DB Schema | 2019_01_16_195644_add_vrf_id_and_bgpLocalAs (131)
> PHP       | 7.2.15-1+ubuntu16.04.1+deb.sury.org+1
> MySQL     | 10.3.12-MariaDB-1:10.3.12+maria~xenial
> RRDTool   | 1.5.5
> SNMP      | NET-SNMP 5.7.3
> ====================================
> 
> [OK]    Composer Version: 1.8.4
> [OK]    Dependencies up-to-date.
> [OK]    Database connection successful
> [OK]    Database schema correct
> [FAIL]  The poller (librenms) has not completed within the last 5 minutes, check the cron job.
> [WARN]  Some devices have not been polled in the last 5 minutes. You may have performance issues.
>         [FIX]:
>         Check your poll log and see: http://docs.librenms.org/Support/Performance/
>         Devices:
>          localhost
>          some IP address
>           and 697 more...
> [FAIL]  Some devices have not completed their polling run in 5 minutes, this will create gaps in data.
>         [FIX]:
>         Check your poll log and see: http://docs.librenms.org/Support/Performance/
>         Devices:
>          ===some IP Address===
> [FAIL]  Discovery has not completed in the last 24 hours.
>         [FIX]:
>         Check the cron job to make sure it is running and using discovery-wrapper.py
> [FAIL]  Some folders have incorrect file permissions, this may cause issues.
>         [FIX]:
>         sudo chown -R librenms:librenms /opt/librenms
>         sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
>         sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
>         Files:
>          /opt/librenms/rrd/10.80.16.133/port-id.rrd
> -------------------------------------------------------------------------------------------------------------------

============daily.sh==========
Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase                        OK
Updating Composer packages                         OK
Updating SQL-Schema                                OK
Updating submodules                                OK
Cleaning up DB                                     OK
Fetching notifications                             OK
Caching PeeringDB data                             OK
=============================

your help is appreciated


#2

I suggest you to update your mysql max connections to at least 500.

Said that. If that happens since you updated your PHP to 7.2, I would start doing a manual ./poller.php -d -h all and check where is hanging out the poller.


#3

Hi!
Try to update max_connection value to 500. and after restart mysql daemon I check the librenms.log the poller stop working :’(

but I notice new log:

[2019-02-11 22:46:46] production.ERROR: SQLSTATE[42000]: Syntax error or access violation: 1066 Not unique table/alias: 'devices' (SQL: SELECT DISTINCT(devices.device_id) FROM devices,locations,devices WHERE (( locations.id = devices.location_id && devices.device_id = devices.device_id ) && ( devices.device_id = devices.device_id )) && devices.device_id=455 AND (locations.location REGEXP "jatinegara" || locations.location REGEXP "ancol" || locations.location REGEXP "pluit" || locations.location REGEXP "buaran" || devices.sysName REGEXP "bks" || locations.location REGEXP "bekasi" || locations.location REGEXP "sunter" || locations.location REGEXP "karawang" ) LIMIT 1) (SQL: SELECT DISTINCT(devices.device_id) FROM devices,locations,devices WHERE (( locations.id = devices.location_id && devices.device_id = devices.device_id ) && ( devices.device_id = devices.device_id )) && devices.device_id=455 AND (locations.location REGEXP "jatinegara" || locations.location REGEXP "ancol" || locations.location REGEXP "pluit" || locations.location REGEXP "buaran" || devices.sysName REGEXP "bks" || locations.location REGEXP "bekasi" || locations.location REGEXP "sunter" || locations.location REGEXP "karawang" ) LIMIT 1)

please advice


#4

Looks like you have an invalid device group rule. Not related to the update at all.

Try group all items in the same table together.


#5

Hi murrant!

I’ve removed the device group rules related the previous issue. Now, the error log not appear :smiley:
But I still struggling with graph issue :frowning:
libre

================./daily.sh===================
[email protected]:/opt/librenms# ./daily.sh
Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase                        OK
Updating Composer packages                         OK
Updated from cf17016 to 25954cc                    OK
Updating SQL-Schema                                OK
Updating submodules                                OK
Cleaning up DB                                     OK
Fetching notifications                             OK
Caching PeeringDB data                             OK


================./validate.php===================
[email protected]:/opt/librenms# ./validate.php
====================================
Component | Version
--------- | -------
LibreNMS  | 1.48.1-32-g25954cc
DB Schema | 2019_01_16_195644_add_vrf_id_and_bgpLocalAs (131)
PHP       | 7.2.15-1+ubuntu16.04.1+deb.sury.org+1
MySQL     | 10.3.12-MariaDB-1:10.3.12+maria~xenial
RRDTool   | 1.5.5
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 1.8.4
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[FAIL]  Discovery has not completed in the last 24 hours.
        [FIX]:
        Check the cron job to make sure it is running and using discovery-wrapper.py

================my.cnf===================
[email protected]:/opt/librenms# cat /etc/mysql/my.cnf
[mysqld]
max_connections         = 500
wait_timeout            = 20000
#interactive_timeut     = 21600
query_cache_size        = 0
query_cache_type        = 0
performance_schema      = ON
join_buffer_size        = 2048
innodb_buffer_pool_size = 8G
innodb_buffer_pool_size = 16M
#general_log_file        = /var/log/mysql/mysql.log
#general_log             = 1
innodb_flush_log_at_trx_commit = 0

please advice :pray:


#6

Hi, seems the issue has been resolved.

The last command i’ve enter:

#systemctl restart rrdcached.service
#apt-get install php7.2-ldap
#apt-get install php7.2-snmp
#systemctl restart apache2


#7

Perfect! Glad you solved it.