Syslog-ng is not retaining messages from localhost

[keeperAndy]

I am using syslog-ng (v3.5.6) on CentOS 7.9 with LibreNMS v21.9.0-10 to capture syslog feeds from clients. The feeds from external clients defined with source s_net work fine, but the localhost’s own logs which should be captured via source s_sys are empty when I go to inspect them in LibreNMS’ Syslog viewer.

I know source s_sys is working because syslog-ng is writing to /var/log/messages with this config added to the recommended one in /etc/syslog-ng/syslog-ng.conf:

destination d_local {
    file("/var/log/messages");
};

log {
        source(s_sys);
        destination(d_local);
};

Yet the same source(s_sys) is not yielding entries to destination(d_librenms) when called in the recommended config:

destination d_librenms {
        program("/opt/librenms/syslog.php" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC||$MSG||$PROGRAM\n") template-escape(yes));
};

log {
        source(s_net);
        source(s_sys);
        destination(d_librenms);
};

Shouldn’t I be seeing the localhost’s own syslog messages when filtering for it in LibreNMS Syslog viewer?

[murrant]

Seems like it should be I wouldn’t suggest sending the server syslogs to LibreNMS. Maybe if you filter them.

[keeperAndy]

I figured out a workaround for anyone interested in seeing the LibreNMS localhost’s syslog feed inside LibreNMS’ Syslog viewer in the GUI.

I enabled debugging with syslog-ng-ctl debug --set=on long enough to capture some traffic and see how it compared to the hosts I’m monitoring via TCP & UDP. I noticed the messages from localhost lacked the $HOST parameter:

Incoming log entry; line='<131>Oct 4 15:05:19 andyTestLocalLog: TESTLOG on Mon Oct 4 15:05:19 EDT 2021'

Contrast with an entry from another host shipping syslog:

Incoming log entry; line='<30>Oct 4 15:00:01 remoteHostname systemd: Started Session 2352 of user root.

This lead me to figure the destination template expecting a $HOST value might break however syslog.php is handling the raw syslog feed. Here is the recommended entry per the docs:

destination d_librenms {
program("/opt/librenms/syslog.php" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC||$MSG||$PROGRAM\n") template-escape(yes));
};

And here is my additional entry to account for the lack of $HOST value on localhost’s syslog stream:

destination d_librelocalhost {
program("/opt/librenms/syslog.php" template ("hostname.mydomain.tld||$FACILITY||$PRIORITY||$LEVEL||$TAG||$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC||$MSG||$PROGRAM\n") template-escape(yes));
};

I’m simply hard-coding the localhost’s FQDN into the template definition so that a value is present for syslog.php to pass to the database. This also requires adjustments to the log definitions in /etc/syslog-ng/syslog-ng.conf:

log {
source(s_net);
destination(d_librenms);
};

log {
source(s_sys);
destination(d_librelocalhost);
};

Simply run syslog-ng-ctl reload to activate the changes and you will start seeing your LibreNMS host’s own syslog in the Syslog Viewer of the GUI.