Oxidized Trying to Backup Nodes That Don't Exist

Hi Team,

We have email alerts setup through Oxidized’s hook feature to send us an email if a node fails to backup. However, this week we’ve started getting failed to backup notices for devices that are no longer in LibreNMS. And, we’re getting a small set of devices that are in LibreNMS that are saying they couldnt be backed up, when they show green in the system.

It’s starting to drive me and my team bananas, and I don’t know where to start for this, as it’s pulling the devices from LibreNMS and the devices don’t exist anymore. Any advice or suggestions is appreciated!

Thanks!

It sounds like it’s running on an old node list. Have you restarted the Oxidized service?

Hi,

I have, yes. It refreshes the nodes, does the backups, then I get another few emails about the nonexistent devices.

Are the devices that are no longer in LibreNMS no longer in Oxidized as well, check the oxidized web interface directly to confirm.

Correct, the devices are no longer listed in the Oxidized menu.

I rebooted the box this morning, as I commented out the lines for the email alerts in the config file, and it was still sending emails, even after a restart of the service.

Just really weird. I’ll update on next scheduled run.

Problem persisted.

I removed the devices from the Oxidized git file, we’ll see if that does anything. The devices don’t show when I run the curl against LibreNMS’s API for this, so I don’t know where it’s getting its devices fed from.

Weirdly enough, I’m still getting email alerts from Oxidized, despite commenting out the hooks line for it in the config file. I’m very confused as to what’s going wrong here, but still digging…

It sounds like maybe your oxidized config isn’t what you expect it to be. Oxidized will attempt to load config from multiple locations Oxidized configuration is in YAML format. Configuration files are subsequently sourced from /etc/oxidized/config then ~/.config/oxidized/config

I’m going to just blow away our oxidized and rebuild it. I found several places where it could be pulling it’s config from (due to the issues we had during initial install) but even after cleaning those up, I still am getting emails (and shouldnt) and still trying to backup non existent nodes so it’s pulling from a config file somewhere I can’t find. I appreciate everyone’s advice!

1 Like

Okay… After a complete rebuild of Oxidized (every single file removed and redone), it’s STILL trying to backup devices that no longer exist… I’m at a loss here guys

Welp… Found the problem… I’m going to cry a river right quick LOL!!

We migrated to a new LibreNMS server… And the old server got turned back ON during a frantic outage… So the old server was sending these emails… NOT the new LibreNMS server…

Please hold while I cry in a corner real fast :rofl: :sneezing_face:

Thanks to everyone for your advice. SMH, I can’t believe this. I thought I was going insane.

1 Like

Haha, least you got it sorted :slight_smile:

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.