Port Page Not Loading - New Install

[jakub]

I have a brand new install (16 vCPU, 32GB RAM) with a single device added. I disabled all heavy pollers such as BGP and ports. This device has about 7,500 ports but ports are only discovered, and not polled (poller disabled). This is so that I can, later on, select which ports I’d like to poll, and poll only those.

Device takes about 3 seconds to poll which is great, but when I click on ‘ports’ tab, it times out, and I see the below in /opt/librenms/logs/librenms. All other pages load pretty must instantly, except maybe Routing page as well which takes a while, or sometimes timesout (there are a lot of VRFs on that device).

[2021-04-16 09:25:31] production.ERROR: Maximum execution time of 30 seconds exceeded {"userId":1,"exception":"[object] (Symfony\\Component\\ErrorHandler\\Error\\FatalError(code: 0): Maximum execution time of 30 seconds exceeded at /opt/librenms/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:54)
[stacktrace]
#0 {main}
"}
[2021-04-16 09:29:20] production.ERROR: Maximum execution time of 30 seconds exceeded {"userId":1,"exception":"[object] (Symfony\\Component\\ErrorHandler\\Error\\FatalError(code: 0): Maximum execution time of 30 seconds exceeded at /opt/librenms/vendor/ezyang/htmlpurifier/library/HTMLPurifier/Config.php:284)
[stacktrace]
#0 {main}
"}
/opt/librenms/discovery.php new 2021-04-16 09:30:01 - 0 devices discovered in 0.009 secs
/opt/librenms/poller.php 1 2021-04-16 09:30:07 - 1 devices polled in 5.054 secs
[2021-04-16 09:30:36] production.ERROR: Maximum execution time of 30 seconds exceeded {"userId":1,"exception":"[object] (Symfony\\Component\\ErrorHandler\\Error\\FatalError(code: 0): Maximum execution time of 30 seconds exceeded at /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/helpers.php:1145980250)
[stacktrace]
#0 {main}
"}

When checking /var/logs/nginx/error.log, I see a lot of below.

2021/04/16 09:27:32 [error] 947#947: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.1, server: nms.example.com, request: "GET /device/1/ports HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "nms.example.com", referrer: "http://nms.example.com/device/1/routing"

I’ve tweaked php-cli and php-fpm but no difference.

~# cat /etc/php/7.4/cli/php.ini | egrep ^'opc|memory_l'
memory_limit = 512M
opcache.enable=1
opcache.enable_cli=1
opcache.memory_consumption=512
opcache.max_accelerated_files=10000

~# cat /etc/php/7.4/fpm/php.ini | egrep ^'opc|memory_l'
memory_limit = 512M
opcache.enable=1
opcache.memory_consumption=512
opcache.max_accelerated_files=10000
opcache.revalidate_freq=200

I have added rrdcached.

||/ Name           Version       Architecture Description
+++-==============-=============-============-=================================
ii  rrdcached      1.7.2-3build2 amd64        data caching daemon for RRDtool

RRDcached section in config.php:

$config['rrdcached']    = "unix:/run/rrdcached.sock";
$config['rrdtool_version'] = '1.7.2';

I ran MySQL Tuner, and changed added recommendations to /etc/mysql/mariadb.conf.d/50-server.cnf.

query_cache_limit       = 1M
query_cache_size        = 0
query_cache_type        = 0
join_buffer_size        = 256K
tmp_table_size          = 16M
max_heap_table_size     = 16M
performance_schema      = ON
innodb_log_file_size    = 16M

Output of validate.php.

    $ ./validate.php
    Cannot load Zend OPcache - it was already loaded
    Cannot load Zend OPcache - it was already loaded
    Cannot load Zend OPcache - it was already loaded
    ====================================
    Component | Version
    --------- | -------
    LibreNMS  | 21.3.0-102-g385ef98ff
    DB Schema | 2021_04_08_151101_add_foreign_keys_to_port_group_port_table (208)
    PHP       | 7.4.3
    Python    | 3.8.5
    MySQL     | 10.3.25-MariaDB-0ubuntu0.20.04.1
    RRDTool   | 1.7.2
    SNMP      | NET-SNMP 5.8
    ====================================

    [OK]    Composer Version: 2.0.12
    [OK]    Dependencies up-to-date.
    [OK]    Database connection successful
    [OK]    Database schema correct
    Cannot load Zend OPcache - it was already loaded
    Cannot load Zend OPcache - it was already loaded
    Cannot load Zend OPcache - it was already loaded
    Cannot load Zend OPcache - it was already loaded
    Cannot load Zend OPcache - it was already loaded
    Cannot load Zend OPcache - it was already loaded
    Cannot load Zend OPcache - it was already loaded
    Cannot load Zend OPcache - it was already loaded
    Cannot load Zend OPcache - it was already loaded
    Cannot load Zend OPcache - it was already loaded
    [WARN]  Your install is over 24 hours out of date, last update: Wed, 14 Apr 2021 15:11:51 +0000
            [FIX]:
            Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.

Any ideas? Is there a timeout that I need to increase? I see it complains about 30 seconds timeout, which might not be enough to load 7500 ports onto a page.