504 Gateway Time-out

When trying to view the network map (Overview → Maps → Network) the browser hangs and then im presented with a 504 Gateway Time-out nginx error.

The map was working earlier in the evening right after adding my first devices and suddenly stopped working

When asking for help and support, please provide as much information as possible. This should include:

[email protected]:~$ ./validate.php

Component Version
LibreNMS 22.10.0-66-g0c365d08e (2022-11-02T19:50:46-04:00)
DB Schema 2022_09_03_091314_update_ports_adsl_table_with_defaults (248)
PHP 8.1.2-1ubuntu2.6
Python 3.10.6
Database MariaDB 10.6.7-MariaDB-2ubuntu1.1
RRDTool 1.7.2
SNMP 5.9.1
===========================================

[OK] Composer Version: 2.4.4
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK]
[OK] Database schema correct
[OK] MySQl and PHP time match
[OK] Active pollers found
[OK] Dispatcher Service not detected
[OK] Locks are functional
[OK] Python poller wrapper is polling
[OK] Redis is unavailable
[OK] rrd_dir is writable
[OK] rrdtool version ok

things ive already done to try and fix the issue:

  • Increased php_memory_limit via lnms config:set php_memory_limit 2048
  • increased pm.maxchildren to 20

everything else seems to be working properly

nginx log:

[email protected]:/opt/librenms# tail -f /var/log/nginx/error.log 
2022/11/02 23:56:21 [error] 971#971: *5239 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.250.100.3, server: librenms.hernandoclerk.org, request: "GET /map HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.hernandoclerk.org"
2022/11/02 23:57:34 [error] 971#971: *5239 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.250.100.3, server: librenms.hernandoclerk.org, request: "GET /map HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.hernandoclerk.org"
2022/11/02 23:59:51 [error] 971#971: *5239 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.250.100.3, server: librenms.hernandoclerk.org, request: "GET /map HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.hernandoclerk.org"
2022/11/03 00:04:46 [error] 971#971: *5486 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.250.100.3, server: librenms.hernandoclerk.org, request: "GET /map HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.hernandoclerk.org"
2022/11/03 00:09:58 [error] 971#971: *5517 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.250.100.3, server: librenms.hernandoclerk.org, request: "GET /map/debug HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.hernandoclerk.org"
2022/11/03 00:15:15 [error] 973#973: *5521 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.250.100.3, server: librenms.hernandoclerk.org, request: "GET /map/debug HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.hernandoclerk.org"
2022/11/03 00:20:54 [error] 491392#491392: *7 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.250.100.3, server: librenms.hernandoclerk.org, request: "GET /map HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.hernandoclerk.org"
2022/11/03 08:40:09 [error] 491390#491390: *805 upstream timed out (110: Unknown error) while reading response header from upstream, client: 192.168.113.150, server: librenms.hernandoclerk.org, request: "GET /map HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.hernandoclerk.org", referrer: "http://librenms.hernandoclerk.org/device/1/overview"
2022/11/03 08:41:15 [error] 491390#491390: *1204 upstream timed out (110: Unknown error) while reading response header from upstream, client: 192.168.113.150, server: librenms.hernandoclerk.org, request: "GET /map HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.hernandoclerk.org", referrer: "http://librenms.hernandoclerk.org/"
2022/11/03 08:47:31 [error] 491390#491390: *1297 upstream timed out (110: Unknown error) while reading response header from upstream, client: 192.168.113.150, server: librenms.hernandoclerk.org, request: "GET /map HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.hernandoclerk.org", referrer: "http://librenms.hernandoclerk.org/"

please help…

tinkering with things still I was able to load a map from a switch neighbors page.

It took about a minute and a half to load however. I think this may be a resource issue but unsure.

double checked the php.ini in FPM and increase the max execution time which seems to have helped.

the network map is EXTREMELY slow to load, like 3 min or more…

any ideas?

Little info about my setup:

I have a bog standard setup and it loads almost instantly for things like this (I have some vis layout customisations, hence the tiers):

image

It gets output from here, but that’s as far as I can help at this point: librenms/print-map.inc.php at 790c4b340269900ffacc0165d35f5387acbc46f6 · librenms/librenms · GitHub

Do the device dependency maps have the same delays?

its working much better now, been continuing to adjust performance. next step is to setup dist pollers bc the CPU utilization on the webserver goes nuts when polling…

here’s my map! it load in less than 2 sec now…