Hi,
I just migrated from a standalone LibreNMS to a distributed solution. There are 5 pollers globally and a single centralized server running rrdcache, rrdtool, memcache and the webui.
Everything looks great from a ./validate.php perspective, but the web UI is very very slow (see output below). Pages are taking about 10 seconds to load. Pretty much any page. Graphs are showing up once the page loads, but the loading is slow. It’s almost unusable.
I have performed the following performance optimizations:
- Removed modules that are not needed (see config.php snippet below)
- Performed recommended mysql configuration change from https://docs.librenms.org/Support/Performance/
- Tuned max repeaters and fping (see config.php snippet below)
- Increased the poller threads to twice the CPU (i.e. 56)
- Stopped using hostnames and just used IP addresses for polling.
Any ideas what else I can try? Or any other areas I should look for improvements?
Thanks in advance for your help with this.
bash-4.2$ ./validate.php
| Component | Version |
|---|---|
| LibreNMS | 1.52-20-g437a577 |
| DB Schema | 2019_02_10_220000_add_dates_to_fdb (390) |
| PHP | 7.2.17 |
| MySQL | 5.7.14-8-log |
| RRDTool | 1.7.1 |
| SNMP | NET-SNMP 5.7.2 |
====================================
[OK] Composer Version: 1.8.6
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[FAIL] Some folders have incorrect file permissions, this may cause issues.
[FIX]:
sudo chown -R librenms:librenms /opt/librenms
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
Files:
/opt/librenms/storage/framework/views/a43cace168384dc4da58e095a70a4ce0
bash-4.2$ ./validate.php -g distributed
| Component | Version |
|---|---|
| LibreNMS | 1.52-20-g437a577 |
| DB Schema | 2019_02_10_220000_add_dates_to_fdb (390) |
| PHP | 7.2.17 |
| MySQL | 5.7.14-8-log |
| RRDTool | 1.7.1 |
| SNMP | NET-SNMP 5.7.2 |
====================================
[OK] Composer Version: 1.8.6
[OK] Dependencies up-to-date.
bash-4.2$
# disable os modules
$config[‘os’][‘ios’][‘poller_modules’][‘cipsec-tunnels’] = false;
$config[‘os’][‘asa’][‘poller_modules’][‘cipsec-tunnels’] = false;
$config[‘os’][‘asa’][‘poller_modules’][‘cisco-ipsec-flow-monitor’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-ace-serverfarms’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-ace-loadbalancer’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-mac-accounting’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-voice’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-remote-access-monitor’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-sla’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-otv’] = false;
$config[‘os’][‘ios’][‘poller_modules’][‘cisco-vpdn’] = false;
$config[‘os’][‘ios’][‘discovery_modules’][‘ciscoo-cef’] = false;
$config[‘os’][‘ios’][‘discovery_modules’][‘cisco-sla’] = false;
$config[‘os’][‘ios’][‘discovery_modules’][‘disco-mac-accounting’] = false;
$config[‘os’][‘ios’][‘discovery_modules’][‘cisco-otv’] = false;
$config[‘os’][‘ios’][‘discovery_modules’][’ cisco-pw’] = false;
$config[‘os’][‘ios’][‘discovery_modules’][‘cisco-vrf’] = false;
$config[‘os’][‘ios’][‘discovery_modules’][‘cisco-vrf-lite’] = false;
$config[‘os’][‘iosxe’][‘poller_modules’][‘cisco-mac-accounting’] = false;
$config[‘os’][‘iosxe’][‘poller_modules’][‘cisco-voice’] = false;
$config[‘os’][‘iosxe’][‘poller_modules’][‘cisco-remote-access-monitor’] = false;
$config[‘os’][‘iosxe’][‘poller_modules’][‘cisco-sla’] = false;
$config[‘os’][‘iosxe’][‘poller_modules’][‘cisco-otv’] = false;
$config[‘os’][‘iosxe’][‘poller_modules’][‘cisco-vpdn’] = false;
$config[‘os’][‘iosxe’][‘poller_modules’][‘cipsec-tunnels’] = false;
$config[‘os’][‘iosxe’][‘discovery_modules’][‘cisco-cef’] = false;
$config[‘os’][‘iosxe’][‘discovery_modules’][‘cisco-sla’] = false;
$config[‘os’][‘iosxe’][‘discovery_modules’][‘cisco-mac-accounting’] = false;
$config[‘os’][‘iosxe’][‘discovery_modules’][‘cisco-otv’] = false;
$config[‘os’][‘iosxe’][‘discovery_modules’][‘cisco-pw’] = false;
$config[‘os’][‘iosxe’][‘discovery_modules’][‘cisco-vrf’] = false;
$config[‘os’][‘iosxe’][‘discovery_modules’][‘cisco-vrf-lite’] = false;
$config[‘poller_modules’][‘ospf’] = false;
$config[‘poller_modules’][‘stp’] = false;
$config[‘poller_modules’][‘ipmi’] = false;
$config[‘discovery_modules’][‘ospf’] = false;
$config[‘discovery_modules’][‘stp’] = false;
$config[‘discovery_modules’][‘ipmi’] = false;
$config[‘poller_modules’][‘cisco-ipsec-flow-monitor’] = false;
$config[‘discovery_modules’][‘cisco-ipsec-flow-monitor’] = false;
# SNMP tuning
$config[‘snmp’][‘max_repeaters’] = 25;
$config[‘snmp’][‘timeout’] = 3;
$config[‘snmp’][‘retries’] = 3;
# ping tuning. Timeout is the time it waits
#count is the number of packets sent to each target
#interva is the time in ms it wasis between the packets
$config[‘fping_options’][‘timeout’] = 500;
$config[‘fping_options’][‘count’] = 10;
$config[‘fping_options’][‘interval’] = 500;