Syslog not working properly


#1

I am a relative NOOB at Linux, but learning…

I had my syslogs up and running fine for months, I am using syslog-ng as my receiver.
All was fine up to a few weeks ago, then syslogs just stopped turning up in the web gui. No configuration changes were made, things just stopped. I can run a tcpdump and see the syslog messages are received by the server, (Centos). I have run daily.sh and validate.php and everything seems good. Checking the database syslog table returns 0 rows…
So, I have to assume there is either something wrong in my syslog.php file or my syslog-ng.conf file?
My headres for syslogs come in like this:

12:20:05.451436 IP 10.10.11.1.syslog > 192.168.2.74.syslog: SYSLOG local4.warning, length: 152
and:
12:19:47.380897 IP 192.168.2.1.syslog > 192.168.2.74.syslog: SYSLOG local4.warning, length: 156
I guess I am not processing the headers properly for the information to actually pass to the database?
I have been trying different settings in the configurations to no avail. If I go back to the base configuration files, can someone point configuration file changes I may need to do?
I have checked the status of the syslog-ng service, it is running. I have restarted it, booted the server, etc, still a no go.
Thanks in advance for any information that can be provided.


#2

Did IP or hostname change on any of those devices ?

Can you post full validate.php and syslog-ng.conf

and output of, getfacl /opt/librenms/syslog.php


#3

I have been playing a bit with the setup, after the issue started.
I did change 1 firewall to FQDN within LibreNMS as a test, and have tried some configuration changes.
Below is the information you have requested`
getfacl: Removing leading ‘/’ from absolute path names

file: opt/librenms/syslog.php

owner: librenms

group: librenms

user::rwx
group::r-x
other::r-x
@version:3.5
@include “scl.conf”

syslog-ng configuration file.

This should behave pretty much like the original syslog on RedHat. But

it could be configured a lot smarter.

See syslog-ng(8) and syslog-ng.conf(5) for more information.

Note: it also sources additional configuration files (*.conf)

located in /etc/syslog-ng/conf.d/

options {
flush_lines(0);
use_dns(no);
use_fqdn(no);
owner(root);
group(adm);
perm(0640);
stats_freq(0);
bad_hostname("^gconfd$");
keep_hostname(yes);
};

source s_sys {
system();
internal();

};

source s_net {
tcp(port(514) flags(syslog-protocol));
udp(port(514) flags(syslog-protocol));
};

########################

Destinations

########################
destination d_librenms {
program("/opt/librenms/syslog.php" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC||$MSG||$PROGRAM\n") template-escape(yes));
};

########################

Log paths

########################
log {
source(s_net);
source(s_sys);
destination(d_librenms);
};

Source additional configuration files (.conf extension only)

@include “/etc/syslog-ng/conf.d/*.conf”

vim:ft=syslog-ng:ai:si:ts=4:sw=4:et:

EDIT:
Removed text of Validate.php for ease of reading the post.

Thanks, very much.


#4

He means run ./validate.php and post the out put not the validate.php code.


#5

sorry…

====================================
Component Version
LibreNMS 1.50-17-g1b46d87
DB Schema 2019_02_10_220000_add_dates_to_fdb (132)
PHP 7.2.16
MySQL 5.5.60-MariaDB
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 1.8.4
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct


#6

Just ran Validate.php again and got this:
====================================

Component Version
LibreNMS 1.50-23-ge7ba1b1
DB Schema 2019_02_10_220000_add_dates_to_fdb (132)
PHP 7.2.16
MySQL 5.5.60-MariaDB
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 1.8.4
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[FAIL] The poller (netmonitor.perillon.com) has not completed within the last 5 minutes, check the cron job.
[FAIL] We have found some files that are owned by a different user than librenms, this will stop you updating automatically and / or rrd files being updated causing graphs to fail.
[FIX]:
sudo chown -R librenms:librenms /opt/librenms
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
Files:
/opt/librenms/.git/ORIG_HEAD


#7

It seems your syslog-ng config is a bit different from the docs.

Try copy config again from https://docs.librenms.org/Extensions/Syslog/

and restart the service


#8

I have just completed this, I will update in an hour or so.
Thanks


#10

Sorry something in my last post went horribly wrong…
That was a no go, is it possible my syslog.php file is corrupted or wrong?

<?php // FIXME : use db functions properly // $device_id_host = @dbFetchCell("SELECT device_id FROM devices WHERE `hostname` = '".mres($entry['host'])."' OR `sysName` = '".mres($entry['host'])."'"); // $device_id_ip = @dbFetchCell("SELECT device_id FROM ipv4_addresses AS A, ports AS I WHERE A.ipv4_address = '" . $entry['host']."' AND I.port_id = A.port_id"); use LibreNMS\Config; function get_cache($host, $value) { global $dev_cache; if (!isset($dev_cache[$host][$value])) { switch ($value) { case 'device_id': // Try by hostname $ip = inet_pton($host); if (inet_ntop($ip) === false) { $dev_cache[$host]['device_id'] = dbFetchCell('SELECT `device_id` FROM devices WHERE `hostname` = ? OR `sysName` = ?', array($host, $host)); } else { $dev_cache[$host]['device_id'] = dbFetchCell('SELECT `device_id` FROM devices WHERE `hostname` = ? OR `sysName` = ? OR `ip` = ?', array($host, $host, $ip)); } // If failed, try by IP if (!is_numeric($dev_cache[$host]['device_id'])) { $dev_cache[$host]['device_id'] = dbFetchCell('SELECT `device_id` FROM `ipv4_addresses` AS A, `ports` AS I WHERE A.ipv4_address = ? AND I.port_id = A.port_id', array($host)); } break; case 'os': $dev_cache[$host]['os'] = dbFetchCell('SELECT `os` FROM devices WHERE `device_id` = ?', array(get_cache($host, 'device_id'))); break; case 'version': $dev_cache[$host]['version'] = dbFetchCell('SELECT `version` FROM devices WHERE `device_id`= ?', array(get_cache($host, 'device_id'))); break; case 'hostname': $dev_cache[$host]['hostname'] = dbFetchCell('SELECT `hostname` FROM devices WHERE `device_id` = ?', array(get_cache($host, 'device_id'))); break; default: return null; }//end switch }//end if return $dev_cache[$host][$value]; }//end get_cache() function process_syslog($entry, $update) { global $config, $dev_cache; foreach ($config['syslog_filter'] as $bi) { if (strpos($entry['msg'], $bi) !== false) { return $entry; } } $entry['host'] = preg_replace("/^::ffff:/", "", $entry['host']); if ($new_host = Config::get("syslog_xlate")[$entry['host']]) { $entry['host'] = $new_host; } $entry['device_id'] = get_cache($entry['host'], 'device_id'); if ($entry['device_id']) { $os = get_cache($entry['host'], 'os'); $hostname = get_cache($entry['host'], 'hostname'); if ((isset($config['enable_syslog_hooks'])) && ($config['enable_syslog_hooks']) && (isset($config['os'][$os]['syslog_hook'])) && (is_array($config['os'][$os]['syslog_hook']))) { foreach ($config['os'][$os]['syslog_hook'] as $k => $v) { $syslogprogmsg = $entry['program'].": ".$entry['msg']; if ((isset($v['script'])) && (isset($v['regex'])) && ((preg_match($v['regex'], $syslogprogmsg)))) { shell_exec(escapeshellcmd($v['script']).' '.escapeshellarg($hostname).' '.escapeshellarg($os).' '.escapeshellarg($syslogprogmsg).' >/dev/null 2>&1 &'); } } } if (in_array($os, array('ios', 'iosxe', 'catos'))) { // multipart message if (strpos($entry['msg'], ':') !== false) { $matches = array(); $timestamp_prefix = '([\*\.]?[A-Z][a-z]{2} \d\d? \d\d:\d\d:\d\d(.\d\d\d)?( [A-Z]{3})?: )?'; $program_match = '(?%?[A-Za-z\d\-_]+(:[A-Z]* %[A-Z\d\-_]+)?)'; $message_match = '(?.*)'; if (preg_match('/^' . $timestamp_prefix . $program_match . ': ?' . $message_match . '/', $entry['msg'], $matches)) { $entry['program'] = $matches['program']; $entry['msg'] = $matches['msg']; } unset($matches); } else { // if this looks like a program (no groups of 2 or more lowercase letters), move it to program if (!preg_match('/[(a-z)]{2,}/', $entry['msg'])) { $entry['program'] = $entry['msg']; unset($entry['msg']); } } } elseif ($os == 'linux' and get_cache($entry['host'], 'version') == 'Point') { // Cisco WAP200 and similar $matches = array(); if (preg_match('#Log: \[(?P.*)\] - (?P.*)#', $entry['msg'], $matches)) { $entry['msg'] = $matches['msg']; $entry['program'] = $matches['program']; } unset($matches); } elseif ($os == 'linux') { $matches = array(); // pam_krb5(sshd:auth): authentication failure; logname=root uid=0 euid=0 tty=ssh ruser= rhost=123.213.132.231 // pam_krb5[sshd:auth]: authentication failure; logname=root uid=0 euid=0 tty=ssh ruser= rhost=123.213.132.231 if (empty($entry['program']) and preg_match('#^(?P([^(:]+\([^)]+\)|[^\[:]+\[[^\]]+\])) ?: ?(?P.*)$#', $entry['msg'], $matches)) { $entry['msg'] = $matches['msg']; $entry['program'] = $matches['program']; } // SYSLOG CONNECTION BROKEN; FD='6', SERVER='AF_INET(123.213.132.231:514)', time_reopen='60' // pam_krb5: authentication failure; logname=root uid=0 euid=0 tty=ssh ruser= rhost=123.213.132.231 // Disabled because broke this: // diskio.c: don't know how to handle 10 request // elseif($pos = strpos($entry['msg'], ';') or $pos = strpos($entry['msg'], ':')) { // $entry['program'] = substr($entry['msg'], 0, $pos); // $entry['msg'] = substr($entry['msg'], $pos+1); // } // fallback, better than nothing... elseif (empty($entry['program']) and !empty($entry['facility'])) { $entry['program'] = $entry['facility']; } unset($matches); } elseif ($os == 'procurve') { $matches = array(); if (preg_match('/^(?P[A-Za-z]+): {2}(?P.*)/', $entry['msg'], $matches)) { $entry['msg'] = $matches['msg']. " [". $entry['program']. "]"; $entry['program'] = $matches['program']; } unset($matches); } elseif ($os == 'zywall') { // Zwwall sends messages without all the fields, so the offset is wrong $msg = preg_replace("/\" /", '";', stripslashes($entry['program'].':'.$entry['msg'])); $msg = str_getcsv($msg, ';'); $entry['program'] = null; foreach ($msg as $param) { list($var, $val) = explode("=", $param); if ($var == 'cat') { $entry['program'] = str_replace('"', '', $val); } } $entry['msg'] = join(" ", $msg); }//end if if (!isset($entry['program'])) { $entry['program'] = $entry['msg']; unset($entry['msg']); } $entry['program'] = strtoupper($entry['program']); $entry = array_map('trim', $entry); if ($update) { dbInsert( array( 'device_id' => $entry['device_id'], 'program' => $entry['program'], 'facility' => $entry['facility'], 'priority' => $entry['priority'], 'level' => $entry['level'], 'tag' => $entry['tag'], 'msg' => $entry['msg'], 'timestamp' => $entry['timestamp'], ), 'syslog' ); } unset($os); }//end if return $entry; }//end process_syslog()

#11

It’s much better if you paste using paste bin and then post a link or use the back tics for code.


#12

You checked $config['enable_syslog'] = 1; is present in config.php ?

I think by default on centos syslog-ng logs go directly into /var/log/messages, can you see them in there?

I believe LibreNMS will only pick up syslogs if the hostname / IP recieved from syslog matches like-for-like in librenms. Check from what hostname/IP you receive in the syslog-ng logs and compare to LibreNMS device overview page.


#13

The config file does have $config[‘enable_syslog’] = 1
/var/log/messages files are all 0 Bytes for messages files.
I have changed 1 device to FQDN under “Devices” the second is by IP address.
Messages come in to:
IP.Address.syslog Could that be the issue?


#14

In /var/log/audit I am seeing a bunch of messages like this:
They seem to be suspect to me, but I have no idea what they mean…

type=AVC msg=audit(1555076964.896:34666694): avc:  denied  { search } for  pid=19777 comm="php" name="cache" dev="dm-0" ino=306416 scontext=system_u:system_r:syslogd_t:s0 tcontext=system_u:object_r:httpd_sys_rw_content_t:s0 tclass=dir permissive=0

type=SYSCALL msg=audit(1555076964.896:34666694): arch=c000003e syscall=21 success=no exit=-13 a0=7ffbc47ae568 a1=0 a2=7ffbbe232a01 a3=62696c2f74706f2f items=0 ppid=22812 pid=19777 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=“php” exe="/usr/bin/php" subj=system_u:system_r:syslogd_t:s0 key=(null)


#15

I have decided to just scrap the “Fix” idea and go for a reinstall.
I am not sure how to close this issue out, as unresolved, but would like to do so.


#16

I know you marked this is as closed since you re-installed, but I had the same issue. Turned out it was selinux blocking syslog-ng from calling the php script. I set my selinux policy to disabled and syslog messages started showing in the GUI again.


#17

I also had problems with syslog not working in LibreNMS. The logs were hitting my server but not getting through to LibreNMS. SELinux was indeed blocking logs for some reason. I used the setenforce Permisive command to change the mode (previously set enforcing). As soon I it was applied logs came through to my LibreNMS. Thanks to John_Shrader for the troubleshooting tip.