504 Gateway Time-out (nginx/1.18.0 (Ubuntu))

Help
1

Hi,

In the last few days I started to experience performance degradation on my system, with every click I get a timeout, has anyone had the same problem? What would be the solution!
Thanks

--------- | -------
LibreNMS  | 24.3.0-24-g19344f058 (2024-04-19T03:45:30+01:00)
DB Schema | 2024_04_10_093513_remove_device_perf (291)
PHP       | 8.2.18
Python    | 3.10.12
Database  | MariaDB 10.6.16-MariaDB-0ubuntu0.22.04.1
RRDTool   | 1.7.2
SNMP      | 5.9.1
===========================================

[OK]    Composer Version: 2.7.2
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    Database and column collations are correct
[OK]    Database schema correct
[OK]    MySQL and PHP time match
[OK]    Active pollers found
[OK]    Dispatcher Service not detected
[OK]    Locks are functional
[OK]    Python poller wrapper is polling
[OK]    Redis is unavailable
[OK]    rrdtool version ok
[OK]    Connected to rrdcached


2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/dash/top-interfaces HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/dash/top-interfaces HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/dash/top-errors HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/dash/generic-graph HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/dash/generic-graph HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/dash/generic-graph HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax_table.php HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/dash/server-stats HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/table/eventlog HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/table/syslog HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:03:07 [error] 1411#1411: *20959 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.22, server: librenms.mydomain.com, request: "POST /ajax_table.php HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/overview?dashboard=1"
2024/04/19 12:03:07 [error] 1411#1411: *20959 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.22, server: librenms.mydomain.com, request: "POST /ajax/dash/server-stats HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/overview?dashboard=1"
2024/04/19 12:03:07 [error] 1411#1411: *20959 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.22, server: librenms.mydomain.com, request: "POST /ajax/table/eventlog HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/overview?dashboard=1"
2024/04/19 12:03:07 [error] 1411#1411: *20959 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.22, server: librenms.mydomain.com, request: "POST /ajax/table/syslog HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/overview?dashboard=1"

504 Gateway Time-out
nginx/1.18.0 (Ubuntu)
2

What is the timeout set to on your reverse proxy?

3

Hi @murrant ,

I don’t use reverse proxy, the service has always worked very well before.!
It is installed on a physical server and is the only service on the machine.
Thanks

4

Oh, likely you have exceeded your php-fpm children. You need to tune that for your usecase.

Check php-fpm logs too.

5

Hi @murrant ,

That logs?

 tail /var/log/php7.4-fpm.log
[18-Aug-2022 09:26:39] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:27:39] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:28:40] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:30:40] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:32:40] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:33:45] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:35:45] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:36:00] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:36:33] NOTICE: Terminating ...
[18-Aug-2022 09:36:33] NOTICE: exiting, bye-bye!


 ps aux | grep fpm
root        1209  0.0  0.0 435868 43252 ?        Ss   Apr21   0:08 php-fpm: master process (/etc/php/8.2/fpm/php-fpm.conf)
www-data    1514  0.0  0.0 436220 16812 ?        S    Apr21   0:00 php-fpm: pool www
www-data    1515  0.0  0.0 436220 16812 ?        S    Apr21   0:00 php-fpm: pool www
librenms 2349970  0.0  0.0 439344 58712 ?        S    14:49   0:01 php-fpm: pool librenms
librenms 2350012  0.0  0.0 439536 60528 ?        S    14:49   0:01 php-fpm: pool librenms
librenms 2351843  0.0  0.0 439208 54788 ?        S    14:50   0:00 php-fpm: pool librenms
root     2843212  0.0  0.0   6620  2236 pts/1    S+   17:37   0:00 grep --color=auto fpm