Woke up this morning to a 502 Bad Gateway for the LibreNMS web ui. Restarted the server, nothing. Absolutely nothing was changed on the server before this started happening. I restored back to a snapshot from a few days ago when the web UI was working - same issue. I’m stumped.
/var/log/nginx/error.log
2025/05/29 17:18:48 [crit] 1351#1351: *3 connect() to unix:/run/php-fpm-librenms.sock failed (2: No such file or directory) while connecting to upstream, client: 10.x.x.x, server: librenms.x.x, request: "GET /alert-schedule HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm-librenms.sock:", host: "librenms.x.x", referrer: "https://librenms.x.x/overview?dashboard=16"
systemctl status php8.3-fpm (same error after bouncing the service)
ERROR: unable to bind listening socket for address '/run/php-fpm-librenms.sock': Permission denied (13)
ERROR: FPM initialization failed
php8.3-fpm.service: Main process exited, code=exited, status=78/CONFIG
php8.3-fpm.service: Failed with result 'exit-code'.
Failed to start php8.3-fpm.service - The PHP 8.3 FastCGI Process Manager.
./validate.php
===========================================
Component | Version
--------- | -------
LibreNMS | 25.5.0-60-g105923b65 (2025-05-29T10:33:49-04:00)
DB Schema | 2025_05_25_183627_drop_view_port_mac_link (341)
PHP | 8.3.6
Python | 3.12.3
Database | MariaDB 10.11.11-MariaDB-0ubuntu0.24.04.2
RRDTool | 1.7.2
SNMP | 5.9.4.pre2
===========================================
[OK] Composer Version: 2.8.9
[OK] Dependencies up-to-date.
[OK] Database Connected
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK] Database and column collations are correct
[OK] Database schema correct
[OK] MySQL and PHP time match
[OK] Active pollers found
[OK] Dispatcher Service not detected
[OK] Locks are functional
[OK] Python poller wrapper is polling
[OK] Redis is unavailable
[OK] rrdtool version ok
[OK] Connected to rrdcached
Yup, I only have a single librenms.conf file in my pool. Here’s my config file (just the pieces from the install docs). I haven’t touched this file since the initial install.
/etc/php/8.3/fpm/pool.d/librenms.conf
user = librenms
group = librenms
listen = /run/php-fpm-librenms.sock
listen.owner = www-data
listen.group = www-data
;listen.mode = 0660
Correct, there is no sock file. I tried to create my own using touch, and chown librenms:librenms and chmod 777, same issue. Also tried pointing config files to the /run/php/ subfolder, then chown librenms:librenms chmod that folder instead. No dice.
MIB search path: /root/.snmp/mibs:/usr/share/snmp/mibs:/usr/share/snmp/mibs/iana:/usr/share/snmp/mibs/ietf
Cannot find module (SNMPv2-MIB): At line 0 in (none)
Cannot find module (IF-MIB): At line 0 in (none)
Cannot find module (IP-MIB): At line 0 in (none)
Cannot find module (TCP-MIB): At line 0 in (none)
Cannot find module (UDP-MIB): At line 0 in (none)
Cannot find module (HOST-RESOURCES-MIB): At line 0 in (none)
Cannot find module (NOTIFICATION-LOG-MIB): At line 0 in (none)
Cannot find module (DISMAN-EVENT-MIB): At line 0 in (none)
Cannot find module (DISMAN-SCHEDULE-MIB): At line 0 in (none)
Cannot find module (HOST-RESOURCES-TYPES): At line 0 in (none)
Cannot find module (MTA-MIB): At line 0 in (none)
Cannot find module (NETWORK-SERVICES-MIB): At line 0 in (none)
Cannot find module (UCD-DISKIO-MIB): At line 0 in (none)
Cannot find module (UCD-DLMOD-MIB): At line 0 in (none)
Cannot find module (LM-SENSORS-MIB): At line 0 in (none)
Cannot find module (UCD-SNMP-MIB): At line 0 in (none)
Cannot find module (UCD-DEMO-MIB): At line 0 in (none)
Cannot find module (SNMP-TARGET-MIB): At line 0 in (none)
Cannot find module (NET-SNMP-AGENT-MIB): At line 0 in (none)
Cannot find module (SNMP-FRAMEWORK-MIB): At line 0 in (none)
Cannot find module (SNMP-MPD-MIB): At line 0 in (none)
Cannot find module (SNMP-USER-BASED-SM-MIB): At line 0 in (none)
Cannot find module (SNMP-VIEW-BASED-ACM-MIB): At line 0 in (none)
Cannot find module (SNMP-COMMUNITY-MIB): At line 0 in (none)
Cannot find module (IPV6-ICMP-MIB): At line 0 in (none)
Cannot find module (IPV6-MIB): At line 0 in (none)
Cannot find module (IPV6-TCP-MIB): At line 0 in (none)
Cannot find module (IPV6-UDP-MIB): At line 0 in (none)
Cannot find module (IP-FORWARD-MIB): At line 0 in (none)
Cannot find module (NET-SNMP-PASS-MIB): At line 0 in (none)
Cannot find module (NET-SNMP-EXTEND-MIB): At line 0 in (none)
Cannot find module (SNMP-NOTIFICATION-MIB): At line 0 in (none)
Cannot find module (SNMPv2-TM): At line 0 in (none)
Cannot find module (NET-SNMP-VACM-MIB): At line 0 in (none)
[30-May-2025 12:21:23] ERROR: unable to bind listening socket for address '/run/php-fpm-librenms.sock': Permission denied (13)
[30-May-2025 12:21:23] ERROR: FPM initialization failed
Ugh, it was indeed AppArmor. I put PHP in complain mode, restarted the service, and the web ui came back instantly.
sudo aa-complain /usr/sbin/php-fpm8.3
I’m running Ubuntu 24.04.2 LTS, perhaps there was a critical update pushed and applied overnight. If anyone else comes across this issue, try the above command and restart php.
