Syslog messages suddenly stopped showing up in GUI

Help
1

Syslog messages were reporting correctly in the LibreNMS GUI until 3/17/19, at which point they simply stopped; however, I’m not aware of any changes to the server.

image
image1353×369 25.1 KB

TCP dump shows the syslog packets coming on the configured port (tcp 601)

[root@utlibrenms librenms]# tcpdump -i any host 10.50.0.10
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
15:05:17.442396 IP 10.50.0.10.60116 > utlibrenms.syslog-conn: Flags [P.], seq 1200648839:1200648951, ack 2193519317, win 115, options [nop,nop,TS val 946734092 ecr 7761440], length 112

And syslog_ng logging for d_local is working correctly
destination d_local {
file(“/var/log/syslog-ng/messages_${HOST}”); };

[root@utlibrenms librenms]# ls -lah /var/log/syslog-ng/
total 436M
drwxr-xr-x. 2 root root 86 Mar 25 12:32 .
drwxr-xr-x. 10 root root 4.0K Mar 25 12:50 …
-rw-r-----. 1 root adm 88M Mar 25 15:04 messages_10.50.0.10
-rw-r-----. 1 root adm 298K Mar 25 15:04 messages_localhost
-rw-r-----. 1 root adm 348M Mar 25 12:32 messages_utlibrenms
[root@utlibrenms librenms]#

Syslog is enabled in config.php:

Enable syslog extension

$config[‘enable_syslog’] = 1;
#$config[‘syslog_purge’] = 30;

Any ideas on why this log is not getting into Librenms? Thank you much.

2

Please also find my /opt/librenms/syslog.php below. Thanks!

image
image1283×505 10.9 KB

3

Is it working for any other host in the LibreNMS GUI like localhost ?

Did you try a reboot, or syslog-ng restart?

Post the full syslog-ng conf file.

4

Hi Chas,

I do not see new syslog message for any host in the GUI. I did already try restarting syslog-ng as well as a server reboot. Please find the the conf file below. Thanks.

[root@utlibrenms syslog-ng]# cat syslog-ng.conf
@version:3.5
@include "scl.conf"

# syslog-ng configuration file.
#
# This should behave pretty much like the original syslog on RedHat. But
# it could be configured a lot smarter.
#
# See syslog-ng(8) and syslog-ng.conf(5) for more information.
#
# Note: it also sources additional configuration files (*.conf)
#       located in /etc/syslog-ng/conf.d/

options {
        chain_hostnames(off);
        flush_lines(0);
        use_dns(no);
        use_fqdn(no);
        owner("root");
        group("adm");
        perm(0640);
        stats_freq(0);
        bad_hostname("^gconfd$");
};


source s_sys {
    system();
    internal();

};

source s_net {
        tcp(port(601) flags(syslog-protocol));
        udp(port(514) flags(syslog-protocol));
};


########################
# Destinations
########################
destination d_librenms {
        program("/opt/librenms/syslog.php" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC||$MSG||$PROGRAM\n") template-escape(yes));
};

destination d_local {
    file("/var/log/syslog-ng/messages_${HOST}"); };

filter f_kernel     { facility(kern); };
filter f_default    { level(debug..emerg) and
                        not (facility(mail)
                        or facility(authpriv)
                        or facility(cron)); };
filter f_auth       { facility(authpriv); };
filter f_mail       { facility(mail); };
filter f_emergency  { level(emerg); };
filter f_news       { facility(uucp) or
                        (facility(news)
                        and level(crit..emerg)); };
filter f_boot   { facility(local7); };
filter f_cron   { facility(cron); };

########################
# Log paths
########################
log {
        source(s_net);
        source(s_sys);
        destination(d_librenms);
        destination (d_local);
};

# Source additional configuration files (.conf extension only)
@include "/etc/syslog-ng/conf.d/*.conf"


# vim:ft=syslog-ng:ai:si:ts=4:sw=4:et:
5

Maybe you reached the bigint limit for id?

select id from eventlog ORDER BY id DESC LIMIT 1;

6

Apologies, I don’t have much experience with mysql, but i get the following:

MariaDB [librenms]> select id from eventlog ORDER BY id DESC LIMIT 1;
ERROR 1054 (42S22): Unknown column 'id' in 'field list'
7

I think this is what you were looking for:

MariaDB [librenms]> select event_id from eventlog ORDER BY event_id DESC LIMIT 1;
+----------+
| event_id |
+----------+
|   112649 |
+----------+
1 row in set (0.01 sec)
8

Event_id is set to init10, so it should be able to go up to 4294967295 i believe

MariaDB [librenms]> show columns from eventlog;
+-----------+------------------+------+-----+---------------------+----------------+
| Field     | Type             | Null | Key | Default             | Extra          |
+-----------+------------------+------+-----+---------------------+----------------+
| event_id  | int(10) unsigned | NO   | PRI | NULL                | auto_increment |
| device_id | int(10) unsigned | YES  | MUL | NULL                |                |
| datetime  | datetime         | NO   | MUL | 1970-01-02 00:00:01 |                |
| message   | text             | YES  |     | NULL                |                |
| type      | varchar(64)      | YES  |     | NULL                |                |
| reference | varchar(64)      | YES  |     | NULL                |                |
| username  | varchar(128)     | YES  |     | NULL                |                |
| severity  | tinyint(4)       | NO   |     | 2                   |                |
+-----------+------------------+------+-----+---------------------+----------------+
9

can you take that line out of conf

so it looks like;

log {
        source(s_net);
        source(s_sys);
        destination(d_librenms);
};

and then restart syslog-ng service

10

Yes, sorry :sweat_smile:

11

I can, but that stops the messages I care about from getting logged, and doesn’t appear to change anything in the GUI.

12

Ok just saw somewhere if you want two destinations you need separate log declarations otherwise it will only send to one of them and not both. Thought worth a shot.

1 Like
13

Hello,

I am having the same issue. No syslog messages from any device are showing up since 4/7. I have checked my syslog-ng.conf and everything looks good. Restarted syslog-ng and the server itself. Have there been any updates or fixes discovered?

2019_04_15_10_47_58_Syslog_LibreNMS
2019_04_15_10_47_58_Syslog_LibreNMS1369×327 14 KB

Thanks,

14

edit: purged all syslog by setting $config[‘syslog_purge’] =1 and running daily.sh. Then optimized the syslog table by running

mysql> optimize table syslog;

The syslog table is now 176k and is not growing but the server is still receiving syslog messages per tcpdump. Any help with this would be appreciated. Thanks.

15

I was able to get syslog working again by switching to the nightly release and updating with daily.sh. As soon as it completed syslog messages started to appear in the GUI.

16

Do you remember what version of librenms you were running on before?

and by nightly release, do you mean the daily dev branch (master), or monthly stable branch (release) ?
#https://docs.librenms.org/General/Releases/#development-branch

17

I had been on the monthly stable so latest version was 1.50. I switched to the daily dev branch (master) by commenting out $config[‘update_channel’] = ‘release’; in config.php and running daily.sh.

Current validate.php:

[librenms@librenms librenms]$ ./validate.php
====================================
Component | Version
--------- | -------
LibreNMS  | 1.50-41-gd069d0f
DB Schema | 2019_02_10_220000_add_dates_to_fdb (132)
PHP       | 7.2.16
MySQL     | 5.5.60-MariaDB
RRDTool   | 1.4.8
SNMP      | NET-SNMP 5.7.2
====================================

[OK]    Composer Version: 1.8.5
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[FAIL]  Discovery has not completed in the last 24 hours.
    [FIX]:
    Check the cron job to make sure it is running and using discovery-wrapper.py
18

In the snapshot you attached, I noticed your Hostname has a dot in it ,

There was a fix two months ago https://github.com/librenms/librenms/commit/06d61fc5288b5b3b8a4fd30df5e210bdf4f7ab11 which suspects that dots broke syslog.

Perhaps the last stable version didn’t update?

Anyhow good that its fixed now :slight_smile:

19

Yes, we do use FQDN in all of our devices, including the libre server itself. If that isn’t best practice we can change it. Odd that the bug would have manifested on the 7th, more than a week after the update to latest stable.

I will have to keep a better eye on it in the future if and when we switch back to stable.

20

I just returned from an extended vacation to find that syslog stopped working again on 4/21. The last thing logged was logrotate starting on the librenms server. I can’t believe it’s just a coincidence. I stopped logrotate and also restarted the server. Syslog is still not working. Does anyone have any further suggestions?

image
image937×243 9.85 KB