RSyslog passthrough to LibreNMS not working after Ubuntu 24.04 Upgrade

[longingtrend]

Hello,

Last friday I’ve updated our Ubuntu 22.04 to 24.04 and got LibreNMS to work again using: Step by Step procedure to update to latest PHP 8.3.11. I’m using Rsyslog.

I’ve noticed though that the Syslog page on the webgui hasn’t been updating since then. I’ve used the LibreNMS documentation to validate the configs.

$config['enable_syslog'] = 1;

Is properly setup. rsyslog.conf does have:

$ModLoad imudp
$UDPServerRun 514

And I have the file /etc/rsyslog.d/30-librenms.conf filled with:

# Feed syslog messages to librenms
module(load="omprog")

template(name="librenms"
        type="string"
        string= "%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timegenerated:8:25%||%msg%||%programname%\n")
        action(type="omprog"
        binary="/opt/librenms/syslog.php"
        template="librenms")

& stop

I also tried changing it to %fromhost-ip% and %hostname%

If I run tcpdump -i eth0 udp port 514 or look at /var/log/syslog the logs gets properly sent to the VM and do show up.

If I check the rsyslog service under the root user, this happens:

action 'action-2-omprog' resumed (module 'omprog') [v8.2312.0 try https://www.rsyslog.com/e/2359 ]
Oct 30 14:45:45  rsyslogd[296197]: omprog: failed to execute program '/opt/librenms/syslog.php': Permission denied

But if I check it under the librenms user everything is green/alright. I also tried giving full permissions or rights to the file for root to test, but didn’t make a difference.

As far as I can tell SELinux is turned off and to test I’ve also turned off AppArmor. But both didn’t help.

The ./validate.php:

===========================================
Component | Version
--------- | -------
LibreNMS  | 24.9.1-83-g6a14a9bd7 (2024-10-30T03:48:25+01:00)
DB Schema | 2024_10_12_210114_custom_map_legend_colours (303)
PHP       | 8.3.6
Python    | 3.12.3
Database  | MariaDB 10.11.8-MariaDB-0ubuntu0.24.04.1
RRDTool   | 1.7.2
SNMP      | 5.9.4.pre2
===========================================

[OK]    Composer Version: 2.8.1
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database connection successful
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    Database and column collations are correct
[OK]    Database schema correct
[OK]    MySQL and PHP time match
[OK]    Active pollers found
[OK]    Dispatcher Service not detected
[OK]    Locks are functional
[OK]    Python poller wrapper is polling
[OK]    Redis is unavailable
[OK]    rrd_dir is writable
[OK]    rrdtool version ok

What else can I check to fix this problem? Many thanks for the answer.

[Roger]

I ran into similar problems when upgrading and it was AppArmor that was the cause for me.
I tried chasing all the paths etc. that Rsyslog was using when integrated with LibreNMS and gave up.

The solution for me was to disable AppArmor for rsyslogd using: sudo aa-disable /path/to/rsyslogd

HTH,
Roger

[longingtrend]

Yes, that seems to be it!

Even though I have explicitly turned off AppArmor with systemctl and rebooted the OS sudo apparmor_status still showed an enforced policy on /usr/sbin/rsyslogd and even though AppArmor was killed it still was doing something.

sudo aa-disable /usr/sbin/rsyslogd fixed the issue, thanks!