Add other vendors to oxidized syslog hook


#1

First time I’ve actually remembered to ask here rather than on github :wink:

Allied Telesis AW+ products will have a syslog log message when the user saves the config in the next software release (5.4.8-2.1) which is due out in I think early Nov.

There will also be a line now (again similar to other vendors) at the top of the config with a time stamp and who last saved the device config.

As such, I want to add support for this enhancement in LibreNMS so when a user saves the config, the syslog hook will cause oxidized to auto login and pull the latest config.

I haven’t got this working yet, and I’m just wondering what I need to do.

Is there a way to debug whether the hook below is actually triggering the script?

$config['os']['awplus']['syslog_hook'][] = Array('regex' => '/IMI\[[0-9]+\]: Startup-config saved on/', 
   'script' => '/opt/librenms/scripts/syslog-notify-oxidized.php');

This is the log message:

2018-09-06T23:08:34+12:00 user.notice c1-x908-stk IMI[1059]: Startup-config saved on Thu Sep  6 23:08:32 2018 +1200 by manager via con0 (ttyS0)                                

And from LibreNMS:

Then in /scripts/syslog-notify-oxidized.php

elseif (preg_match('/IMI\[[0-9]+\]:\sStartup configuration saved by (?P<user.+)+ via ().*/', $msg, $matches)) {
    oxidized_node_update($hostname, $msg, $matches['user']);

I’m aware above won’t work as it doesn’t skip the timestamp in the log message (this is new), but I’m first of all just trying to figure out whether the initial hook is actually working. Thoughts on how?

Thanks


Juniper ScreenOS Syslog Hook
#2

So I had another look at this today, and managed to get it to work from a simple point of view.

It seems librenms/oxidized doesn’t like the regex on IMI\[[0-9]+\]:\s. But can match fine on '/Startup-config saved on/'

Looking at the RegEx with the former, I can’t see a reason why this wouldn’t work.

Where I’m confused is whether or not the matching is done before or after the logs are shown in LibreNMS.

As per the screenshot above, there is no PID listed under the program.

I checked with wireshark, and below is the full string which comes out of the syslog message. From this we can see the PID is included with the message sent to syslog-ng:

2018-09-17T16:48:11+12:00 c1-x908-stk IMI[1091]: Startup-config saved on Mon Sep 17 16:48:10 2018 +1200 by manager via con0 (ttyS0)

Regardless, I think I’ve come up with a pretty good alternative:

'/IMI.+.Startup-config saved on/'

Instead of trying to match on the variable amount of numbers in the PID, simply match on IMI following by anything until “Startup-config” etc.

Ok I’ve figured it out…

It appears the PID does in fact get stripped at some point. We can see this in the message which is passed to Oxidized:

Hopefully my ramblings above might be useful for someone else if they try and add support for another vendor…


#3

For anyone else trying to add other vendor support, here are some thoughts:

  1. For some reason, I couldn’t test Regex changes in either config.php or ./scripts/syslog-notify-oxidized.php without fully rebooting the server.
  • I verified this simply through the fact that disabling the syslog hook through $config['enable_syslog_hooks'] = 0 did not result in the hook against devices being stopped. I also killed oxidized and restarted it, but still had the hook working…Hence rebooting the server being the only way I could figure this out…
  1. Verifying the hook is working is a two step process.

Verify config.php configuration is working.

The easiest way I’ve found to do this, is simply point 'script' => to a different script than /opt/librenms/scripts/syslog-notify-oxidized.php.

In my case, I did this through:

$config['os']['awplus']['syslog_hook'][] = Array('regex' => '/IMI.+.Startup-config saved on/', 'script' => '/opt/librenms/z.sh');

All z.sh does is touch a file called ZWORKS when the script is called:

 cat z.sh 
#!/bin/sh

touch ZWORKS
[email protected]:~$ chmod +x z.sh 

So after doing this and rebooting, then triggering the hook by saving a device’s config, I checked to see if ZWORKS file exists in /opt/librenms.

Verify /scripts/syslog-notify-oxidized.php works

After confirming config.php is working, followed by pointing it back at the above script, the next step is to verify the log is being passed correctly to oxidized.

Probably the best thing here is just to try add configuration for your vendor/device to /scripts/syslog-notify-oxidized.php, and then start oxidized with debug enabled.

e.g. oxidized -d

As a side note, if you have issues killing oxidized in debug mode, try the following

  • ctrl + c, then ctrl + z
  • kill -9 $(cat /home/oxidized/.config/oxidized/pid)

I’d also recommend having a dev box for this, so you don’t touch your production oxidized/librenms box.

It can also be useful to turn oxidized debug on, and watch the device’s logs using tail -f /home/oxidized/.config/oxidized/logs/<device>-ssh

Hope that helps someone :slight_smile:

Thanks,

Matt