Syslog data not being consumed by LibreNMS

Hello,
I’ve identified a problem where my LibreNMS server isn’t importing syslog messages. My LibreNMS install is from the OVA image provided. The version is 1.51. I’ve narrowed down the issue as much as possible, and I’m not sure how to proceed forward.

From my troubleshooting steps I have confirmed that the syslog messages are being collected by Syslog-NG, but they aren’t being imported into LibreNMS.

If I modify syslog-ng.conf to dump the logs to a file, and then import them using the command below then the messages are successfully imported.

cat /var/log/custom-debug | /opt/librenms/syslog.php

If I change the configuration of syslog-ng.conf back to the default then the messages are not imported. From what I can tell the messages are not being consumed by /opt/librenms/syslog.php, but I don’t know how to verify this.

Output from Validate.php:

[[email protected] librenms]# ./validate.php 
====================================
Component | Version
--------- | -------
LibreNMS  | 1.51
DB Schema | 2019_02_10_220000_add_dates_to_fdb (132)
PHP       | 7.2.17
MySQL     | 5.5.60-MariaDB
RRDTool   | 1.4.8
SNMP      | NET-SNMP 5.7.2
====================================

[OK]    Composer Version: 1.8.5
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[[email protected] librenms]# 

Output from config.php

[[email protected] librenms]# cat config.php
<?php

## Have a look in includes/defaults.inc.php for examples of settings you can set here. DO NOT EDIT defaults.inc.php!

### Database config
$config['db_host'] = 'localhost';
$config['db_user'] = 'librenms';
$config['db_pass'] = '<REMOVED>';
$config['db_name'] = 'librenms';

// This is the user LibreNMS will run as
//Please ensure this user is created and has the correct permissions to your install
$config['user'] = 'librenms';

### This should *only* be set if you want to *force* a particular hostname/port
### It will prevent the web interface being usable form any other hostname
$config['base_url']        = "/";

### Enable this to use rrdcached. Be sure rrd_dir is within the rrdcached dir
### and that your web server has permission to talk to rrdcached.
#$config['rrdcached']    = "unix:/var/run/rrdcached.sock";

### Default community
$config['snmp']['community'] = array('public');

### Authentication Model
$config['auth_mechanism'] = "mysql"; # default, other options: ldap, http-auth
#$config['http_auth_guest'] = "guest"; # remember to configure this user if you use http-auth

### List of RFC1918 networks to allow scanning-based discovery
#$config['nets'][] = "10.0.0.0/8";
#$config['nets'][] = "172.16.0.0/12";
#$config['nets'][] = "192.168.0.0/16";

# Uncomment the next line to disable daily updates
#$config['update'] = 0;

# Number in days of how long to keep old rrd files. 0 disables this feature
$config['rrd_purge'] = 0;

# Uncomment to submit callback stats via proxy
#$config['callback_proxy'] = "hostname:port";

# Set default port association mode for new devices (default: ifIndex)
#$config['default_port_association_mode'] = 'ifIndex';

# Enable the in-built billing extension
$config['enable_billing'] = 1;

# Enable the in-built services support (Nagios plugins)
$config['show_services'] = 1;
$config["fping"] = "/usr/sbin/fping";
$config["rrdcached"] = "unix:/var/run/rrdcached/rrdcached.sock";
$config["update_channel"] = "release";
$config["enable_syslog"] = 1;
$config['snmptraps']['eventlog'] = 'unhandled';
[[email protected] librenms]# 

A snippet from syslog-ng.conf

########################
# Destinations
########################
destination d_librenms {
        program("/opt/librenms/syslog.php" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC||$MSG||$PROGRAM\n") template-escape(yes));
};

#destination d_librenms {
#    file ("/var/log/custom-debug" );
#    file ("/var/log/custom-debug" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC||$MSG||$PROGRAM\n") );
#    file("/var/log/custom-template" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC||$MSG||$PROGRAM\n") template-escape(yes));
#};


########################
# Log paths
########################
log {
        source(s_net);
        source(s_sys);
        destination(d_librenms);
};

As you can see from syslog-ng.conf that I’ve tried several destination variations, and all of the file destinations work as intended. I can then import those file logs into LibreNMS using the cat command mentioned above.

I can confirm that syslog.php is running by using ps.

[[email protected] librenms]# ps aux | grep syslog.php
root     14938  0.0  1.0 370640 19580 ?        R    20:50   0:00 php /opt/librenms/syslog.php
root     14940  0.0  0.0 112708   988 pts/1    R+   20:50   0:00 grep --color=auto syslog.php
root     32259  0.0  0.1 117016  3704 pts/0    S+   May18   0:00 nano syslog.php
[[email protected] librenms]# 

I can also confirm that syslog.php is being called by using systemctl:

[[email protected] librenms]# systemctl status syslog-ng
● syslog-ng.service - System Logger Daemon
   Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2019-05-18 23:22:57 UTC; 21h ago
     Docs: man:syslog-ng(8)
 Main PID: 28424 (syslog-ng)
   CGroup: /system.slice/syslog-ng.service
           ├─19116 php /opt/librenms/syslog.php
           └─28424 /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid

May 18 23:22:57 localhost.localdomain systemd[1]: Starting System Logger Daemon...
May 18 23:22:57 localhost.localdomain systemd[1]: Started System Logger Daemon.
[[email protected] librenms]# 

This is the content of my syslog.php, which is standard as far as I can tell.

[[email protected] librenms]# cat syslog.php 
#!/usr/bin/env php
<?php

/**
 * LibreNMS
 *
 *   This file is part of LibreNMS.
 *
 * @package    LibreNMS
 * @subpackage syslog
 * @copyright  (C) 2006 - 2012 Adam Armstrong
 *
 */

$init_modules = array();
require __DIR__ . '/includes/init.php';

$i = "1";

$s = fopen('php://stdin', 'r');
while ($line = fgets($s)) {
    #logfile($line);
    list($entry['host'],$entry['facility'],$entry['priority'], $entry['level'], $entry['tag'], $entry['timestamp'], $entry['msg'], $entry['program']) = explode("||", trim($line));
    process_syslog($entry, 1);
    unset($entry);
    unset($line);
    $i++;
}
[[email protected] librenms]# 

I can also verify that the syslog entries are being fed into syslog.php:

[[email protected] librenms]# syslog-ng-ctl stats
SourceName;SourceId;SourceInstance;State;Type;Number
dst.program;d_librenms#0;/opt/librenms/syslog.php;a;dropped;0
dst.program;d_librenms#0;/opt/librenms/syslog.php;a;processed;6429324
dst.program;d_librenms#0;/opt/librenms/syslog.php;a;stored;0
destination;d_librenms;;a;processed;6429324
src.internal;s_sys#2;;a;processed;4664206
src.internal;s_sys#2;;a;stamp;1558299450
center;;received;a;processed;4664206
center;;queued;a;processed;6429324
src.none;;;a;processed;0
src.none;;;a;stamp;0
global;payload_reallocs;;a;processed;219147
global;sdata_updates;;a;processed;4
global;msg_clones;;a;processed;0
source;s_sys;;a;processed;4664206
source;s_net;;a;processed;0
[[email protected] librenms]# 

Here is a sample of the syslog messages generated from saving syslog-ng to a file:

192.168.2.1||local4||info||info||a6||2019-05-18 16:18:18||%ASA-6-302015: Built outbound UDP connection 34242847 for Outside:68.169.143.222/23882 (68.169.143.222/23882) to Green:192.168.2.155/23507 (47.208.96.103/23507)||
192.168.2.1||local4||info||info||a6||2019-05-18 16:18:18||%ASA-6-302015: Built outbound UDP connection 34242848 for Outside:82.36.60.155/27353 (82.36.60.155/27353) to Green:192.168.2.155/23507 (47.208.96.103/23507)||
192.168.2.1||local4||info||info||a6||2019-05-18 16:18:18||%ASA-6-302015: Built outbound UDP connection 34242849 for Outside:158.174.102.103/28925 (158.174.102.103/28925) to Green:192.168.2.155/23507 (47.208.96.103/23507)||
192.168.2.1||local4||info||info||a6||2019-05-18 16:18:18||%ASA-6-302015: Built outbound UDP connection 34242850 for Outside:178.88.121.60/27007 (178.88.121.60/27007) to Green:192.168.2.155/23507 (47.208.96.103/23507)||
192.168.2.1||local4||info||info||a6||2019-05-18 16:18:18||%ASA-6-302015: Built outbound UDP connection 34242851 for Outside:79.120.124.29/30317 (79.120.124.29/30317) to Green:192.168.2.155/23507 (47.208.96.103/23507)||
192.168.2.1||local4||info||info||a6||2019-05-18 16:18:18||%ASA-6-302015: Built outbound UDP connection 34242852 for Outside:80.225.66.107/33853 (80.225.66.107/33853) to Green:192.168.2.155/23507 (47.208.96.103/23507)||

If I cat the content of my custom syslogs saves from syslog-ng.conf saved to files then the logs are imported, but running syslog.php as a program being called by syslog-ng seems to result from the logs not being imported? What additional debugging can I do? How can I find out what is going on?

Did you manage to get it working ?

Sounds like a permission issue, since syslog-ng just can’t write to that file, but you have got it working the other way.

Can you post the full syslog-ng.conf file