504 Gateway Time-out (nginx/1.18.0 (Ubuntu))

Hi,

In the last few days I started to experience performance degradation on my system, with every click I get a timeout, has anyone had the same problem? What would be the solution!
Thanks

--------- | -------
LibreNMS  | 24.3.0-24-g19344f058 (2024-04-19T03:45:30+01:00)
DB Schema | 2024_04_10_093513_remove_device_perf (291)
PHP       | 8.2.18
Python    | 3.10.12
Database  | MariaDB 10.6.16-MariaDB-0ubuntu0.22.04.1
RRDTool   | 1.7.2
SNMP      | 5.9.1
===========================================

[OK]    Composer Version: 2.7.2
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    Database and column collations are correct
[OK]    Database schema correct
[OK]    MySQL and PHP time match
[OK]    Active pollers found
[OK]    Dispatcher Service not detected
[OK]    Locks are functional
[OK]    Python poller wrapper is polling
[OK]    Redis is unavailable
[OK]    rrdtool version ok
[OK]    Connected to rrdcached

2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/dash/top-interfaces HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/dash/top-interfaces HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/dash/top-errors HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/dash/generic-graph HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/dash/generic-graph HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/dash/generic-graph HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax_table.php HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/dash/server-stats HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/table/eventlog HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:02:16 [error] 1411#1411: *19818 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.31, server: librenms.mydomain.com, request: "POST /ajax/table/syslog HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/"
2024/04/19 12:03:07 [error] 1411#1411: *20959 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.22, server: librenms.mydomain.com, request: "POST /ajax_table.php HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/overview?dashboard=1"
2024/04/19 12:03:07 [error] 1411#1411: *20959 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.22, server: librenms.mydomain.com, request: "POST /ajax/dash/server-stats HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/overview?dashboard=1"
2024/04/19 12:03:07 [error] 1411#1411: *20959 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.22, server: librenms.mydomain.com, request: "POST /ajax/table/eventlog HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/overview?dashboard=1"
2024/04/19 12:03:07 [error] 1411#1411: *20959 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.6.6.22, server: librenms.mydomain.com, request: "POST /ajax/table/syslog HTTP/2.0", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock", host: "librenms.mydomain.com", referrer: "https://librenms.mydomain.com/overview?dashboard=1"

504 Gateway Time-out
nginx/1.18.0 (Ubuntu)

What is the timeout set to on your reverse proxy?

Hi @murrant ,

I don’t use reverse proxy, the service has always worked very well before.!
It is installed on a physical server and is the only service on the machine.
Thanks

Oh, likely you have exceeded your php-fpm children. You need to tune that for your usecase.

Check php-fpm logs too.

Hi @murrant ,

That logs?

 tail /var/log/php7.4-fpm.log
[18-Aug-2022 09:26:39] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:27:39] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:28:40] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:30:40] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:32:40] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:33:45] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:35:45] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:36:00] WARNING: [pool librenms] server reached pm.max_children setting (5), consider raising it
[18-Aug-2022 09:36:33] NOTICE: Terminating ...
[18-Aug-2022 09:36:33] NOTICE: exiting, bye-bye!

 ps aux | grep fpm
root        1209  0.0  0.0 435868 43252 ?        Ss   Apr21   0:08 php-fpm: master process (/etc/php/8.2/fpm/php-fpm.conf)
www-data    1514  0.0  0.0 436220 16812 ?        S    Apr21   0:00 php-fpm: pool www
www-data    1515  0.0  0.0 436220 16812 ?        S    Apr21   0:00 php-fpm: pool www
librenms 2349970  0.0  0.0 439344 58712 ?        S    14:49   0:01 php-fpm: pool librenms
librenms 2350012  0.0  0.0 439536 60528 ?        S    14:49   0:01 php-fpm: pool librenms
librenms 2351843  0.0  0.0 439208 54788 ?        S    14:50   0:00 php-fpm: pool librenms
root     2843212  0.0  0.0   6620  2236 pts/1    S+   17:37   0:00 grep --color=auto fpm

Hi,

my new settings based in aval ram;

pm.max_children = 201
pm.start_servers = 50
pm.min_spare_servers = 50
pm.max_spare_servers = 150
pm.max_request = 2000

After changed the error disappeared, i already restarted the php.

 [08-May-2024 18:32:14] NOTICE: Terminating ...
[08-May-2024 18:32:14] NOTICE: exiting, bye-bye!
[08-May-2024 18:32:14] NOTICE: fpm is running, pid 149700
[08-May-2024 18:32:14] NOTICE: ready to handle connections
[08-May-2024 18:32:14] NOTICE: systemd monitor interval set to 10000ms

Thanks

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.