Add other vendors to oxidized syslog hook

mattie47 · 6 September 2018 11:24

First time I’ve actually remembered to ask here rather than on github

Allied Telesis AW+ products will have a syslog log message when the user saves the config in the next software release (5.4.8-2.1) which is due out in I think early Nov.

There will also be a line now (again similar to other vendors) at the top of the config with a time stamp and who last saved the device config.

As such, I want to add support for this enhancement in LibreNMS so when a user saves the config, the syslog hook will cause oxidized to auto login and pull the latest config.

I haven’t got this working yet, and I’m just wondering what I need to do.

Is there a way to debug whether the hook below is actually triggering the script?

$config['os']['awplus']['syslog_hook'][] = Array('regex' => '/IMI\[[0-9]+\]: Startup-config saved on/', 
   'script' => '/opt/librenms/scripts/syslog-notify-oxidized.php');

This is the log message:

2018-09-06T23:08:34+12:00 user.notice c1-x908-stk IMI[1059]: Startup-config saved on Thu Sep  6 23:08:32 2018 +1200 by manager via con0 (ttyS0)

And from LibreNMS:

Then in /scripts/syslog-notify-oxidized.php

elseif (preg_match('/IMI\[[0-9]+\]:\sStartup configuration saved by (?P<user.+)+ via ().*/', $msg, $matches)) {
    oxidized_node_update($hostname, $msg, $matches['user']);

I’m aware above won’t work as it doesn’t skip the timestamp in the log message (this is new), but I’m first of all just trying to figure out whether the initial hook is actually working. Thoughts on how?

Thanks

mattie47 · 17 September 2018 05:46

So I had another look at this today, and managed to get it to work from a simple point of view.

It seems librenms/oxidized doesn’t like the regex on IMI\[[0-9]+\]:\s. But can match fine on '/Startup-config saved on/'

Looking at the RegEx with the former, I can’t see a reason why this wouldn’t work.

Where I’m confused is whether or not the matching is done before or after the logs are shown in LibreNMS.

As per the screenshot above, there is no PID listed under the program.

I checked with wireshark, and below is the full string which comes out of the syslog message. From this we can see the PID is included with the message sent to syslog-ng:

2018-09-17T16:48:11+12:00 c1-x908-stk IMI[1091]: Startup-config saved on Mon Sep 17 16:48:10 2018 +1200 by manager via con0 (ttyS0)

Regardless, I think I’ve come up with a pretty good alternative:

'/IMI.+.Startup-config saved on/'

Instead of trying to match on the variable amount of numbers in the PID, simply match on IMI following by anything until “Startup-config” etc.

Ok I’ve figured it out…

It appears the PID does in fact get stripped at some point. We can see this in the message which is passed to Oxidized:

Hopefully my ramblings above might be useful for someone else if they try and add support for another vendor…

mattie47 · 17 September 2018 06:18

For anyone else trying to add other vendor support, here are some thoughts:

For some reason, I couldn’t test Regex changes in either config.php or ./scripts/syslog-notify-oxidized.php without fully rebooting the server.

I verified this simply through the fact that disabling the syslog hook through $config['enable_syslog_hooks'] = 0 did not result in the hook against devices being stopped. I also killed oxidized and restarted it, but still had the hook working…Hence rebooting the server being the only way I could figure this out…

Verifying the hook is working is a two step process.

Verify config.php configuration is working.

The easiest way I’ve found to do this, is simply point 'script' => to a different script than /opt/librenms/scripts/syslog-notify-oxidized.php.

In my case, I did this through:

$config['os']['awplus']['syslog_hook'][] = Array('regex' => '/IMI.+.Startup-config saved on/', 'script' => '/opt/librenms/z.sh');

All z.sh does is touch a file called ZWORKS when the script is called:

 cat z.sh 
#!/bin/sh

touch ZWORKS
librenms@librenmsdev:~$ chmod +x z.sh

So after doing this and rebooting, then triggering the hook by saving a device’s config, I checked to see if ZWORKS file exists in /opt/librenms.

Verify /scripts/syslog-notify-oxidized.php works

After confirming config.php is working, followed by pointing it back at the above script, the next step is to verify the log is being passed correctly to oxidized.

Probably the best thing here is just to try add configuration for your vendor/device to /scripts/syslog-notify-oxidized.php, and then start oxidized with debug enabled.

e.g. oxidized -d

As a side note, if you have issues killing oxidized in debug mode, try the following

ctrl + c, then ctrl + z
kill -9 $(cat /home/oxidized/.config/oxidized/pid)

I’d also recommend having a dev box for this, so you don’t touch your production oxidized/librenms box.

It can also be useful to turn oxidized debug on, and watch the device’s logs using tail -f /home/oxidized/.config/oxidized/logs/<device>-ssh

Hope that helps someone

Thanks,

Matt

tslytsly · 28 January 2019 15:06

@mattie47
Thanks for this.
It never occurred to me to restart the whole server.
But I could not work out why the hooks were not working!

It seems that you must reboot the server when any change is made.

@laf this cannot be working as intended?
Is there a service we should be restarting instead? Rather than the whole server?

kzaykov · 21 March 2019 12:16

Hey!
I am having same issue. Syslog hook is not being triggered:

bash-4.2$ cat config.php | grep syslog

$config['enable_syslog'] = 1;
$config['enable_syslog_hooks'] = 1;
#$config['os']['arista_eos']['syslog_hook'][] = Array('regex' =&gt; '/%SYS-5-CONFIG_I/', 'script' =&gt; '/opt/librenms/scripts/syslog-notify-oxidized.php');
$config['os']['arista_eos']['syslog_hook'][] = Array('regex' =&gt; '/%SYS-5-CONFIG_I/', 'script' =&gt; '/opt/librenms/scripts/z.sh');

Do I need to restart LibreNMS?

TheGreatDoc · 21 March 2019 15:19

No, LibreNMS doesnt have a “restart”.

Are that code really look as the paste?

kzaykov · 21 March 2019 17:53

seems like formatting issue, in fact it is ok:

z.sh is the same dummy script by @mattie47:

bash-4.2$ cat z.sh
#!/bin/sh
touch ZWORKS
bash-4.2$ ls -la z.sh
-rwxr-xr-x 1 librenms librenms 24 Mar 21 12:03 z.sh

regex is correct for Arista EOS, here is what I get in syslog:

%SYS-5-CONFIG_I: Configured from console by kzaykov on vty3 (10.128.128.128)

tried also with different OS, same result.

TheGreatDoc · 22 March 2019 06:45

Your others syslog hooks works?

kzaykov · 22 March 2019 08:43

no, it’s my first try

kzaykov · 25 March 2019 12:42

Any guesses?
There should be a way to trace it I hope

kzaykov · 27 September 2019 10:08

Is there a service we should be restarting instead? Rather than the whole server?

The answer is

TheGreatDocToday at 1:05 PM

@kostas you only need to reload syslog daemon for new hooks, not a server reboot

Thanks all