Syslog broken after deleting and re-adding device with the same IP addres

Hi All,

I seem to have run into an interesting problem with the syslog feature in LibreNMS.

Today I’ve replaced a switch and the new switch is taking the same IP address as the original switch.

In the past I’ve just let LibreNMS rediscover the new switch on the same IP addres and update the model number/interfaces etc, however the issue with doing that is that you can end up with some confused historical statistics/graphs etc since port assignments are no longer the same etc…also old syslog entries that don’t make sense in the context of a completely different model of switch.

So this time I thought I would simply delete the old device from LibreNMS to blow away any remnants of data saved for the old switch and re-add the same IP address as a new device.

This worked fine except for one thing - syslog no longer seems to work for this device, eg Logs->Logging->Syslog just says “No results found!” despite the fact that the new switch is configured to send syslog messages.

I have rsyslog configured on ubuntu server with the recommended reporting plugin for LibreNMS and this has been working great and continues to work for other devices.

At first I thought it might be an issue with the new switch however I checked /var/log/syslog - which is also configured to log everything network devices send to the LibreNMS server via syslog and I can see syslog messages from that particular switch (along with other switches) so the messages are definitely being received by rsyslog, logged, and I see no reason why the LibreNMS plugin wouldn’t be passing them on to LibreNMS as well.

When I deleted the old device and added the new one with the same IP address it now has a different device ID in LibreNMS - the only thing I can think of is that there is still some hidden database link in LibreNMS that is causing syslog messages from that IP address to be funnelled through to a now non-existent device, hence it doesn’t show up in the logs section of the new device that has taken on that IP address.

As I don’t really know how the logging/database in LibreNMS works I’m at a loss how to troubleshoot this issue. Anyone have any ideas ?

Try restarting your syslog daemon… that is what runs the librenms syslog script.

Hi,

Before I got a chance to try this I had to reboot the whole server for other reasons and logging for that device seems to be working again, so I don’t know what the cause was.

I can’t see why restarting rsyslog would help to be honest as it simply passes on the syslog messages to LibreNMS via syslog.php without doing any filtering of its own - it doesn’t treat messages from different devices any differently from each other, it’s LibreNMS which is parsing the messages and deciding which device to associate the messages with.

No changes were made to rsyslog configuration etc at any point… the only thing that was done was the device was deleted in LibreNMS then a different switch with the same IP address was added back in.

I guess it will remain a mystery. :slight_smile:

rsyslog and syslog-ng actually start and monitor the syslog.php script. If you make changes, they won’t be reflected by syslog.php because it is kept running persistently by the syslog daemon.

Ok, I didn’t notice that syslog.php was a persistent child of rsyslog, and ps confirms that.

However I still don’t understand why this would cause the issue I saw - I haven’t made any changes to the rsyslog configuration.

Are you suggesting that every time a new device is added in LibreNMS which is capable of sending syslog messages that the rsyslog service has to be restarted before this will work ?

I’ve never needed to do this before, nor have I seen such a requirement documented before.

So I’m still confused as to how restarting rsyslog would be necessary after adding a new device.