SOLVED (syslog sender timestamp in the future): My Alert Notification won't go away

Help
1

Hello,

I have an alert rule, that looks like this:

"rules": [
    {
        "id": "32",
        "device_id": "-1",
        "rule": "%syslog.timestamp >= %macros.past_5m && %syslog.level = \"crit\"     ",
        "severity": "critical",
        "extra": "{\"mute\":false,\"count\":\"36\",\"delay\":300,\"invert\":false,\"interval\":3600}",
        "disabled": "0",
        "name": "critical syslog message received",
        "query": "SELECT * FROM syslog WHERE (syslog.device_id = ?) && (syslog.timestamp >= (DATE_SUB(NOW(),INTERVAL 5 MINUTE))  &&  syslog.level = \"crit\"     )",
        "proc": ""
    }
]

Now 5 days ago my syslog received a “critical” syslog from a switch and Libre threw an alert notification.
This was the only “critical” syslog message I have received from that switch since then (in fact, there’s no other critical at all).

To my understanding… my check means that every time alert.php runs
it will query for syslog entries of level “crit”, sent by my device and which have a timestamp, that is larger or equal than the timestamp we had 5 minutes ago…

I would expect that I if I did not receive another syslog message of that kind, my alert would change to state 0 (OK) and disappear from my Alerts/Notifications view…

but it does not…:

    {
        "hostname": "172.18.45.100",
        "id": "5658",
        "device_id": "143",
        "rule_id": "32",
        "state": "1",
        "alerted": "1",
        "open": "0",
        "timestamp": "2017-08-12 00:27:06",
        "severity": "critical"
    },

Actually, I have seen this worked as expected for other "crit"s I received from other devices… those alerts disapperaed after 5 minutes… but not on this particular device…

What is the reason for this?
What can I change to make this alert go away?

Any help is appreciated…

Thank you!
-awaum

2

Alerts go away once the SQL query that is run no longer returns data.

In your output you’ve posted your mysql query, run that with the device id for the device alerting:

SELECT * FROM syslog WHERE (syslog.device_id = ?) && (syslog.timestamp >= (DATE_SUB(NOW(),INTERVAL 5 MINUTE)) && syslog.level = "crit" );

3

@laf Thank you for your reply!

The problem was a wrong time configuration on the switch. The date in the syslog message was already for next year… the query on the DB made more visible to me.

Thanks for your help!

4

To get around the problem that devices send an invalid timestamp, I changed the logging behavior on my LibreNMS (which was built from the VM):

In /etc/syslog-ng/syslog-ng.conf I changed the date-strings which are fed into the database to the date when the message was received, not the timestamp provided in the syslog message:

destination d_librenms {
program(“/opt/librenms/syslog.php” template (“$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC||$MSG||$PROGRAM\n”) template-escape(yes));
};

Tha “R_” in the date-components changes this logging behavior from the “S_” date components which are used by default.