LibreNMS Distributed Setup Web GUI is Slow

Hi,

I just migrated from a standalone LibreNMS to a distributed solution. There are 5 pollers globally and a single centralized server running rrdcache, rrdtool, memcache and the webui.

Everything looks great from a ./validate.php perspective, but the web UI is very very slow (see output below). Pages are taking about 10 seconds to load. Pretty much any page. Graphs are showing up once the page loads, but the loading is slow. It’s almost unusable.

I have performed the following performance optimizations:

  1. Removed modules that are not needed (see config.php snippet below)
  2. Performed recommended mysql configuration change from https://docs.librenms.org/Support/Performance/
  3. Tuned max repeaters and fping (see config.php snippet below)
  4. Increased the poller threads to twice the CPU (i.e. 56)
  5. Stopped using hostnames and just used IP addresses for polling.

Any ideas what else I can try? Or any other areas I should look for improvements?
Thanks in advance for your help with this.

bash-4.2$ ./validate.php

Component Version
LibreNMS 1.52-20-g437a577
DB Schema 2019_02_10_220000_add_dates_to_fdb (390)
PHP 7.2.17
MySQL 5.7.14-8-log
RRDTool 1.7.1
SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 1.8.6
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[FAIL] Some folders have incorrect file permissions, this may cause issues.
[FIX]:
sudo chown -R librenms:librenms /opt/librenms
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
Files:
/opt/librenms/storage/framework/views/a43cace168384dc4da58e095a70a4ce0

bash-4.2$ ./validate.php -g distributed

Component Version
LibreNMS 1.52-20-g437a577
DB Schema 2019_02_10_220000_add_dates_to_fdb (390)
PHP 7.2.17
MySQL 5.7.14-8-log
RRDTool 1.7.1
SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 1.8.6
[OK] Dependencies up-to-date.
bash-4.2$

# disable os modules
$config[‘os’][‘ios’][‘poller_modules’][‘cipsec-tunnels’] = false;
$config[‘os’][‘asa’][‘poller_modules’][‘cipsec-tunnels’] = false;
$config[‘os’][‘asa’][‘poller_modules’][‘cisco-ipsec-flow-monitor’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-ace-serverfarms’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-ace-loadbalancer’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-mac-accounting’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-voice’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-remote-access-monitor’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-sla’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-otv’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-vpdn’] = false;
$config[‘os’][‘ios’][‘discovery_modules’][‘ciscoo-cef’] = false;
$config[‘os’][‘ios’][‘discovery_modules’][‘cisco-sla’] = false;
$config[‘os’][‘ios’][‘discovery_modules’][‘disco-mac-accounting’] = false;
$config[‘os’][‘ios’][‘discovery_modules’][‘cisco-otv’] = false;
$config[‘os’][‘ios’][‘discovery_modules’][’ cisco-pw’] = false;
$config[‘os’][‘ios’][‘discovery_modules’][‘cisco-vrf’] = false;
$config[‘os’][‘ios’][‘discovery_modules’][‘cisco-vrf-lite’] = false;
$config[‘os’][‘iosxe’][‘poller_modules’][‘cisco-mac-accounting’] = false;
$config[‘os’][‘iosxe’][‘poller_modules’][‘cisco-voice’] = false;
$config[‘os’][‘iosxe’][‘poller_modules’][‘cisco-remote-access-monitor’] = false;
$config[‘os’][‘iosxe’][‘poller_modules’][‘cisco-sla’] = false;
$config[‘os’][‘iosxe’][‘poller_modules’][‘cisco-otv’] = false;
$config[‘os’][‘iosxe’][‘poller_modules’][‘cisco-vpdn’] = false;
$config[‘os’][‘iosxe’][‘poller_modules’][‘cipsec-tunnels’] = false;
$config[‘os’][‘iosxe’][‘discovery_modules’][‘cisco-cef’] = false;
$config[‘os’][‘iosxe’][‘discovery_modules’][‘cisco-sla’] = false;
$config[‘os’][‘iosxe’][‘discovery_modules’][‘cisco-mac-accounting’] = false;
$config[‘os’][‘iosxe’][‘discovery_modules’][‘cisco-otv’] = false;
$config[‘os’][‘iosxe’][‘discovery_modules’][‘cisco-pw’] = false;
$config[‘os’][‘iosxe’][‘discovery_modules’][‘cisco-vrf’] = false;
$config[‘os’][‘iosxe’][‘discovery_modules’][‘cisco-vrf-lite’] = false;
$config[‘poller_modules’][‘ospf’] = false;
$config[‘poller_modules’][‘stp’] = false;
$config[‘poller_modules’][‘ipmi’] = false;
$config[‘discovery_modules’][‘ospf’] = false;
$config[‘discovery_modules’][‘stp’] = false;
$config[‘discovery_modules’][‘ipmi’] = false;
$config[‘poller_modules’][‘cisco-ipsec-flow-monitor’] = false;
$config[‘discovery_modules’][‘cisco-ipsec-flow-monitor’] = false;

# SNMP tuning
$config[‘snmp’][‘max_repeaters’] = 25;
$config[‘snmp’][‘timeout’] = 3;
$config[‘snmp’][‘retries’] = 3;

# ping tuning. Timeout is the time it waits
#count is the number of packets sent to each target
#interva is the time in ms it wasis between the packets
$config[‘fping_options’][‘timeout’] = 500;
$config[‘fping_options’][‘count’] = 10;
$config[‘fping_options’][‘interval’] = 500;

Have you tried increasing the php memory limit? https://docs.librenms.org/Support/Configuration/#php-settings

Hi Kevin,

Good point. I forgot to mention that. I made this change $config[‘php_memory_limit’] = 1024; for the pollers, and the webui server.

This ended up being a slow query in the database. We added indexes. I don’t know if that will impact the upgrade process.