Hello,
I’ve identified a problem where my LibreNMS server isn’t importing syslog messages. My LibreNMS install is from the OVA image provided. The version is 1.51. I’ve narrowed down the issue as much as possible, and I’m not sure how to proceed forward.
From my troubleshooting steps I have confirmed that the syslog messages are being collected by Syslog-NG, but they aren’t being imported into LibreNMS.
If I modify syslog-ng.conf to dump the logs to a file, and then import them using the command below then the messages are successfully imported.
cat /var/log/custom-debug | /opt/librenms/syslog.php
If I change the configuration of syslog-ng.conf back to the default then the messages are not imported. From what I can tell the messages are not being consumed by /opt/librenms/syslog.php, but I don’t know how to verify this.
Output from Validate.php:
[root@localhost librenms]# ./validate.php
====================================
Component | Version
--------- | -------
LibreNMS | 1.51
DB Schema | 2019_02_10_220000_add_dates_to_fdb (132)
PHP | 7.2.17
MySQL | 5.5.60-MariaDB
RRDTool | 1.4.8
SNMP | NET-SNMP 5.7.2
====================================
[OK] Composer Version: 1.8.5
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[root@localhost librenms]#
Output from config.php
[root@localhost librenms]# cat config.php
<?php
## Have a look in includes/defaults.inc.php for examples of settings you can set here. DO NOT EDIT defaults.inc.php!
### Database config
$config['db_host'] = 'localhost';
$config['db_user'] = 'librenms';
$config['db_pass'] = '<REMOVED>';
$config['db_name'] = 'librenms';
// This is the user LibreNMS will run as
//Please ensure this user is created and has the correct permissions to your install
$config['user'] = 'librenms';
### This should *only* be set if you want to *force* a particular hostname/port
### It will prevent the web interface being usable form any other hostname
$config['base_url'] = "/";
### Enable this to use rrdcached. Be sure rrd_dir is within the rrdcached dir
### and that your web server has permission to talk to rrdcached.
#$config['rrdcached'] = "unix:/var/run/rrdcached.sock";
### Default community
$config['snmp']['community'] = array('public');
### Authentication Model
$config['auth_mechanism'] = "mysql"; # default, other options: ldap, http-auth
#$config['http_auth_guest'] = "guest"; # remember to configure this user if you use http-auth
### List of RFC1918 networks to allow scanning-based discovery
#$config['nets'][] = "10.0.0.0/8";
#$config['nets'][] = "172.16.0.0/12";
#$config['nets'][] = "192.168.0.0/16";
# Uncomment the next line to disable daily updates
#$config['update'] = 0;
# Number in days of how long to keep old rrd files. 0 disables this feature
$config['rrd_purge'] = 0;
# Uncomment to submit callback stats via proxy
#$config['callback_proxy'] = "hostname:port";
# Set default port association mode for new devices (default: ifIndex)
#$config['default_port_association_mode'] = 'ifIndex';
# Enable the in-built billing extension
$config['enable_billing'] = 1;
# Enable the in-built services support (Nagios plugins)
$config['show_services'] = 1;
$config["fping"] = "/usr/sbin/fping";
$config["rrdcached"] = "unix:/var/run/rrdcached/rrdcached.sock";
$config["update_channel"] = "release";
$config["enable_syslog"] = 1;
$config['snmptraps']['eventlog'] = 'unhandled';
[root@localhost librenms]#
A snippet from syslog-ng.conf
########################
# Destinations
########################
destination d_librenms {
program("/opt/librenms/syslog.php" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC||$MSG||$PROGRAM\n") template-escape(yes));
};
#destination d_librenms {
# file ("/var/log/custom-debug" );
# file ("/var/log/custom-debug" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC||$MSG||$PROGRAM\n") );
# file("/var/log/custom-template" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC||$MSG||$PROGRAM\n") template-escape(yes));
#};
########################
# Log paths
########################
log {
source(s_net);
source(s_sys);
destination(d_librenms);
};
As you can see from syslog-ng.conf that I’ve tried several destination variations, and all of the file destinations work as intended. I can then import those file logs into LibreNMS using the cat command mentioned above.
I can confirm that syslog.php is running by using ps.
[root@localhost librenms]# ps aux | grep syslog.php
root 14938 0.0 1.0 370640 19580 ? R 20:50 0:00 php /opt/librenms/syslog.php
root 14940 0.0 0.0 112708 988 pts/1 R+ 20:50 0:00 grep --color=auto syslog.php
root 32259 0.0 0.1 117016 3704 pts/0 S+ May18 0:00 nano syslog.php
[root@localhost librenms]#
I can also confirm that syslog.php is being called by using systemctl:
[root@localhost librenms]# systemctl status syslog-ng
● syslog-ng.service - System Logger Daemon
Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2019-05-18 23:22:57 UTC; 21h ago
Docs: man:syslog-ng(8)
Main PID: 28424 (syslog-ng)
CGroup: /system.slice/syslog-ng.service
├─19116 php /opt/librenms/syslog.php
└─28424 /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid
May 18 23:22:57 localhost.localdomain systemd[1]: Starting System Logger Daemon...
May 18 23:22:57 localhost.localdomain systemd[1]: Started System Logger Daemon.
[root@localhost librenms]#
This is the content of my syslog.php, which is standard as far as I can tell.
[root@localhost librenms]# cat syslog.php
#!/usr/bin/env php
<?php
/**
* LibreNMS
*
* This file is part of LibreNMS.
*
* @package LibreNMS
* @subpackage syslog
* @copyright (C) 2006 - 2012 Adam Armstrong
*
*/
$init_modules = array();
require __DIR__ . '/includes/init.php';
$i = "1";
$s = fopen('php://stdin', 'r');
while ($line = fgets($s)) {
#logfile($line);
list($entry['host'],$entry['facility'],$entry['priority'], $entry['level'], $entry['tag'], $entry['timestamp'], $entry['msg'], $entry['program']) = explode("||", trim($line));
process_syslog($entry, 1);
unset($entry);
unset($line);
$i++;
}
[root@localhost librenms]#
I can also verify that the syslog entries are being fed into syslog.php:
[root@localhost librenms]# syslog-ng-ctl stats
SourceName;SourceId;SourceInstance;State;Type;Number
dst.program;d_librenms#0;/opt/librenms/syslog.php;a;dropped;0
dst.program;d_librenms#0;/opt/librenms/syslog.php;a;processed;6429324
dst.program;d_librenms#0;/opt/librenms/syslog.php;a;stored;0
destination;d_librenms;;a;processed;6429324
src.internal;s_sys#2;;a;processed;4664206
src.internal;s_sys#2;;a;stamp;1558299450
center;;received;a;processed;4664206
center;;queued;a;processed;6429324
src.none;;;a;processed;0
src.none;;;a;stamp;0
global;payload_reallocs;;a;processed;219147
global;sdata_updates;;a;processed;4
global;msg_clones;;a;processed;0
source;s_sys;;a;processed;4664206
source;s_net;;a;processed;0
[root@localhost librenms]#
Here is a sample of the syslog messages generated from saving syslog-ng to a file:
192.168.2.1||local4||info||info||a6||2019-05-18 16:18:18||%ASA-6-302015: Built outbound UDP connection 34242847 for Outside:68.169.143.222/23882 (68.169.143.222/23882) to Green:192.168.2.155/23507 (47.208.96.103/23507)||
192.168.2.1||local4||info||info||a6||2019-05-18 16:18:18||%ASA-6-302015: Built outbound UDP connection 34242848 for Outside:82.36.60.155/27353 (82.36.60.155/27353) to Green:192.168.2.155/23507 (47.208.96.103/23507)||
192.168.2.1||local4||info||info||a6||2019-05-18 16:18:18||%ASA-6-302015: Built outbound UDP connection 34242849 for Outside:158.174.102.103/28925 (158.174.102.103/28925) to Green:192.168.2.155/23507 (47.208.96.103/23507)||
192.168.2.1||local4||info||info||a6||2019-05-18 16:18:18||%ASA-6-302015: Built outbound UDP connection 34242850 for Outside:178.88.121.60/27007 (178.88.121.60/27007) to Green:192.168.2.155/23507 (47.208.96.103/23507)||
192.168.2.1||local4||info||info||a6||2019-05-18 16:18:18||%ASA-6-302015: Built outbound UDP connection 34242851 for Outside:79.120.124.29/30317 (79.120.124.29/30317) to Green:192.168.2.155/23507 (47.208.96.103/23507)||
192.168.2.1||local4||info||info||a6||2019-05-18 16:18:18||%ASA-6-302015: Built outbound UDP connection 34242852 for Outside:80.225.66.107/33853 (80.225.66.107/33853) to Green:192.168.2.155/23507 (47.208.96.103/23507)||
If I cat the content of my custom syslogs saves from syslog-ng.conf saved to files then the logs are imported, but running syslog.php as a program being called by syslog-ng seems to result from the logs not being imported? What additional debugging can I do? How can I find out what is going on?