Alerts stopped working

Help
1

Hi,
I was using LibreNMS for monitoring community mesh without issues until a week ago.
Then suddenly we stopped receiving email and Rocket chat alerts.
I have searched around you PR-s and googled but no solution has been found.

I first ran validate.php and it found a couple of issues with DB, most likely due to the power outage and sudden shutdown.

I backed up the installation and repaired database with commands from validate.php and run it again to verify that no errors have been found.
Then I manually started daily.sh to ensure that latest code is used and rerun validate.php.
No errors have been returned.
validate.php returns this info:

Component Version
LibreNMS 6a79d44
DB Schema 192
PHP 7.0.18-0ubuntu0.16.04.1
MySQL 10.0.29-MariaDB-0ubuntu0.16.04.1
RRDTool 1.5.5
SNMP NET-SNMP 5.7.3

[OK] Database connection successful
[OK] Database schema correct

After that I manually run alerts.php -d and output that into log file.
It outputs this:
https://gist.github.com/robimarko/5e573d18ccc88e325076ba0ba5732dc2
I have tested mail -s manually and it is able to send emails, also rocket chat works when test button is used.
It looks like alert is not issued to transport according to eventlog.

I have exhausted my knowledge so far.

2

Running alerts.php with -d requires you to stop it running via cron first and then waiting for the interval time before running it again.

For the device that should have this alert, run the alerts capture under edit device -> captures. It will show if the rule will have been triggered, if it isn’t triggered then it’s because the rule isn’t matching the data.

3

Do you use any ' or " signs in your alert templates? If yes, you should remove them because they could cause a PHP parse error in an evil eval() function.

If you (after disabling the cron job) run ./alerts.php and don’t get an end timestamp message like End : Wed, 31 May 2017 09:25:31 +0200 this is usually the case. You can then test your template with ./scripts/test-template.php and there you will get the PHP error message.

4

Thank you.
I have stopped the cronjob and then after starting alerts.php -d I see the end mark and start mark.
I have also tested my alert template which and it outputs no errors nor it uses any quotes.
Here is my alert template:
%title {if}
Node name: %sysName
Location: %location
Severity: %severity
Rule: {if %name}%name{else}%rule{/if}

Timestamp: %timestamp{if %state == 0}
Time elapsed: %elapsed{/if}
{if %transport != hipchat}
Unique-ID: %uid{/if}
{if %faults}Faults:
{foreach %faults} #%key: %value.string
{/foreach}{/if}

{if %transport == mail}%hostname{else}%hostname{/if}

I have tried to run alerts under capture and it outputs that and alert has been sent to transport mail.
But under eventlog I dont see it.

I really dont get it that nothing is acctually issued to be sent

5

This {if} seems to be the problem, with your template I get

librenms@mon02:~$ ./scripts/test-template.php -t 1 -h 84 -r 12
PHP Parse error:  syntax error, unexpected ')' in /opt/librenms/includes/alerts.inc.php(433) : eval()'d code on line 1
6

Thanks,it looks like that template was issue.
It is weird since i tested it and no issue was output.

Do you maybe know how instead of hostname in title I can use sysname?

7

this should work %sysName

8

Yes,but how can I use it in subject of emails?
Title uses hostname

9

I modified it in the alerts.php file

Originally it said $obj['title'] = 'Alert for device '.$device['hostname'].' - '.($alert['name'] ? $alert['name'] : $alert['rule']);

Had to add a line above also
sname = explode(’.’,$device[‘sysName’]);

10

Awesome,that is great.
Only issue is that then It can be updated to latest code

11

I found a ghetto way around but probably not the best option. I just back it up rewrite over then copy back over… Im not sure if anyone else has a better solution. All ears

12

Have you tried modifying the alert template title?

Capture.PNG495×786 22.8 KB

13

Yes,I have.
As soon as I enter %sysName there alerts stop getting sent to transports.

Also,anyone knows how to clear all alerts.
Since alerts were not working for about a week I am now getting lot of backlogged alerts

14