Oxidized Issues

Hi, first time posting and not fluent in Linux so bear with me.

I have inherited an environment which has a single VM running Ubuntu which hosts LibreNMS and Oxidized.
It was running fine until recently, Oxidized is unable to pull any config backups and reloading the nodes within the Oxidized Tools section fails as well.
“We couldn’t retrieve the device information from Oxidized”.
“An error occurred while reloading the Oxidized nodes list”.
We had performed OS patching to Ubuntu earlier in June, so I restored the VM from before the patching but it still had the same errors which was surprising.

I’ve tried changing the user within the oxidized.service config from “oxidized” to “root” but it made no difference.

root@inf-librenms:/home/administrator# systemctl status oxidized
● oxidized.service - Oxidized - Network Device Configuration Backup Tool
Loaded: loaded (/lib/systemd/system/oxidized.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2023-07-05 11:05:48 CDT; 30min ago
Process: 944129 ExecStart=/usr/local/bin/oxidized (code=exited, status=1/FAILURE)
Main PID: 944129 (code=exited, status=1/FAILURE)

Jul 05 11:05:48 inf-librenms systemd[1]: Started Oxidized - Network Device Configuration Backup Tool.
Jul 05 11:05:48 inf-librenms oxidized[944129]: I, [2023-07-05T11:05:48.876590 #944129] INFO – : Oxidized starting, running as pid 944129
Jul 05 11:05:48 inf-librenms oxidized[944129]: I, [2023-07-05T11:05:48.878004 #944129] INFO – : lib/oxidized/nodes.rb: Loading nodes
Jul 05 11:05:48 inf-librenms oxidized[944129]: F, [2023-07-05T11:05:48.878162 #944129] FATAL – : Oxidized crashed, crashfile written in /home/oxidized/crash
Jul 05 11:05:48 inf-librenms oxidized[944129]: No such file or directory @ rb_sysopen - /root/.config/oxidized/router.db
Jul 05 11:05:48 inf-librenms systemd[1]: oxidized.service: Main process exited, code=exited, status=1/FAILURE
Jul 05 11:05:48 inf-librenms systemd[1]: oxidized.service: Failed with result ‘exit-code’.

root@inf-librenms:/home/administrator# which oxidized
/usr/local/bin/oxidized

Gems:
oxidized (0.28.0, 0.26.3)
oxidized-script (0.6.0)
oxidized-web (0.13.1)

root@inf-librenms:/home/administrator# cat /lib/systemd/system/oxidized.service
#For debian 8 put it in /lib/systemd/system/
#To set OXIDIZED_HOME instead of the default:
#~${oxidized_user}/.config/oxidized in debian 8, then uncomment
#(and modify as required) the “Environment” variable below so
#systemd sets the correct environment. Tested only on Debian 8.8.
#YMMV otherwise.

#For RHEL / CentOS 7 put it in /etc/systemd/system/
#and call it with systemctl start oxidized.service

[Unit]
Description=Oxidized - Network Device Configuration Backup Tool
After=network-online.target multi-user.target
Wants=network-online.target

[Service]
ExecStart=/usr/local/bin/oxidized
User=root
KillSignal=SIGKILL
Environment=“OXIDIZED_HOME=/home/oxidized”

[Install]
WantedBy=multi-user.target

Does this output imply my router.db file is missing?
root@inf-librenms:/home/administrator# ls ~/.config/oxidized/
config crash logs

Any help is greatly appreciated :slight_smile: Thank you

Jul 05 11:05:48 inf-librenms oxidized[944129]: No such file or directory @ rb_sysopen - /root/.config/oxidized/router.db

That’s telling you that your config file for oxidised is saying its source for devices is /root/.config/oxidized/router.db which doesn’t exist. Only you will know if that’s the correct path or if you are using something else for the device source like the LibreNMS API. But updating the oxidised config file to correct that should help you.

I found that the config file it seems to be using is a default incorrect one under the root directory.
I did find the proper former config file elsewhere under /home/oxidized/.config/oxidized/config.

When I’m trying to start the service, I assume it is trying to start it as “root” user instead of “oxidized” user; how can I correct that so it starts automatically under the correct user?

Change it here in the file you mentioned above

When I first checked that service config, it did have “oxidized” listed as the user. I tried changing it to root but found no success so I changed it back.
So for some reason it is still trying to use the root config instead of the oxidized config

Delete the root config (or at least move it out of the way)

Alright, renamed that config file, rebooted, then found the oxidized service to be in “masked” state.
Manually unmasked it, then started it, it stayed running for more seconds than usual then crashed.

root@inf-librenms:/home/administrator# systemctl status oxidized
● oxidized.service - Oxidized - Network Device Configuration Backup Tool
Loaded: loaded (/lib/systemd/system/oxidized.service; disabled; vendor preset: enabled)
Active: active (running) since Thu 2023-07-06 11:07:28 CDT; 9s ago
Main PID: 6148 (oxidized)
Tasks: 2 (limit: 9413)
Memory: 36.6M
CGroup: /system.slice/oxidized.service
└─6148 /usr/bin/ruby2.7 /usr/local/bin/oxidized

Jul 06 11:07:28 inf-librenms systemd[1]: Started Oxidized - Network Device Configuration Backup Tool.

root@inf-librenms:/home/administrator# systemctl status oxidized
● oxidized.service - Oxidized - Network Device Configuration Backup Tool
Loaded: loaded (/lib/systemd/system/oxidized.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2023-07-06 11:07:49 CDT; 15s ago
Process: 6148 ExecStart=/usr/local/bin/oxidized (code=exited, status=1/FAILURE)
Main PID: 6148 (code=exited, status=1/FAILURE)

Jul 06 11:07:49 inf-librenms oxidized[6148]: from /var/lib/gems/2.7.0/gems/oxidized-0.28.0/lib/oxidized/core.rb:14:in new' Jul 06 11:07:49 inf-librenms oxidized[6148]: from /var/lib/gems/2.7.0/gems/oxidized-0.28.0/lib/oxidized/core.rb:14:in initialize’
Jul 06 11:07:49 inf-librenms oxidized[6148]: from /var/lib/gems/2.7.0/gems/oxidized-0.28.0/lib/oxidized/core.rb:4:in new' Jul 06 11:07:49 inf-librenms oxidized[6148]: from /var/lib/gems/2.7.0/gems/oxidized-0.28.0/lib/oxidized/core.rb:4:in new’
Jul 06 11:07:49 inf-librenms oxidized[6148]: from /var/lib/gems/2.7.0/gems/oxidized-0.28.0/lib/oxidized/cli.rb:13:in run' Jul 06 11:07:49 inf-librenms oxidized[6148]: from /var/lib/gems/2.7.0/gems/oxidized-0.28.0/bin/oxidized:8:in <top (required)>’
Jul 06 11:07:49 inf-librenms oxidized[6148]: from /usr/local/bin/oxidized:23:in load' Jul 06 11:07:49 inf-librenms oxidized[6148]: from /usr/local/bin/oxidized:23:in
Jul 06 11:07:49 inf-librenms systemd[1]: oxidized.service: Main process exited, code=exited, status=1/FAILURE
Jul 06 11:07:49 inf-librenms systemd[1]: oxidized.service: Failed with result ‘exit-code’.

I’d start from scratch, basic config file from the oxidized examples and get it working as a default setup before customising it.

Thank you for your assistance

One of my colleagues fixed our issue; root cause was the API token between LibreNMS and Oxidized was gone. Not sure if it expired or got deleted, but re-creating it fixed the issue with Oxidized.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.