Inherited oxidized setup stopped working

In early January 2019 I inherited a CentOS 7 server running librenms for network monitoring and oxidized for switch config backups. All seemed well until Feb. 5, when oxidized stoped working I had not made any changes in the interim, but since have only updated the server (monthly) with CentOS system updates. We weren’t making any changes to the switch configs so fixing was on a back burner, but now I need to get it working. There seem to be several different problems reported in the oxidized log files, so I’m not sure where to start. Running validate.php on the server comes back with everything ok, but says the “install is more than 24 hours out of date, last updated 23 June 2019”. When I run the daily.sh script I get no errors, but running validate again still says the install was last updated 23 June 2019. librenms appears to be otherwise working fine, though.

I copied the oxidized config file into the librenms Validate YAML screen, and I get this message:

Tags support is not enabled. Enable the “Yaml::PARSE_CUSTOM_TAGS” flag to use “!ruby/regexp” at line 11 (near “prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/”).

Here is the config file, credentials redacted:

----------------------------------------------- BEGIN CONFIG FILE -------------------------------------------------------
username: #########
password: #########
model: boss
interval: 43200
log: /home/oxidized/.config/oxidized/log
use_syslog: true
debug: true
threads: 30
timeout: 20
retries: 3
prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/
pid: /home/oxidized/.config/oxidized/pid
rest: 172.16.10.249:8888
groups: {}
input:
default: telnet, ssh
debug: true
ssh:
secure: false
output:
default: git
git:
user: Oxidized
email: [email protected]
repo: “/opt/oxidized/configs/devices.git”
source:
default: http
debug: true
http:
url: https://itnms.umpublishing.org/api/v0/oxidized
delimiter: !ruby/regexp /:confused:
scheme: https
secure: false
map:
name: hostname
model: os
headers:
X-Auth-Token: 77f42d56b30fe3599f999b0d3b034b27
model_map:
avaya-ers: boss
avaya-vsp: voss
ciscosb: ciscosmb
email_output:
type: exec
events: [post_store, node_fail]
cmd: ‘/home/oxidized/extra/oxidized-report-git-commits | mail -s “Oxidized updates for ${OX_NODE_NAME}” [email protected]
async: true
timeout: 120
------------------------------------------ END CONFIG FILE ----------------------------------------------------------

I have never used ruby, php or git, but I do know my way around linux. Any help would be greatly appreaciated.

I just saw the message regarding the daily.sh on this site, so that explains why my daily.sh hasn’t been running!

Oxidized is actually standalone software it just happens to have nice integration with librenms so unfortunately you won’t gain much from running validate etc

What do your oxidized service logs say, or your log file ?

is oxidizied web working and just the fetching of config from devices failing?

What version are you running ?
gem list | grep oxidized

Where are the oxidized service logs located? I was referring to the librenms logs in my original post.

Thanks for the reply.
I am not sure what oxidized web is, so not sure if it’s working. :-\

In librenms url, the Tools -> Oxidized ->Node List shows nothing. Is that what you mean by oxidized web or is there another url?

gem list|grep oxidized just returns the prompt. gem list returns a list that does not contain oxidized.

you meant librenms?

I would suggest to re-build oxidized or follow install guides and then the librenms implementation docs to check everything is as it should be :slight_smile:

The gem list command will need to be run under the user which oxidized was installed under.

I hate to have to start over from scratch, so many places I can mess things up! But that thought did cross my mind…

I just now tried starting the service (systemctl start oxidized.service). A crash file was created which has as the first line

757: unexpected token at ’

Then it’s complaining about a parse error at the end.

’ [JSON::ParserError]

/usr/share/ruby/json/common.rb:155:in parse' /usr/share/ruby/json/common.rb:155:inparse’
/home/oxidized/.gem/ruby/gems/oxidized-0.24.0/lib/oxidized/source/http.rb:41:in load' /home/oxidized/.gem/ruby/gems/oxidized-0.24.0/lib/oxidized/nodes.rb:15:inblock in load’
/home/oxidized/.gem/ruby/gems/oxidized-0.24.0/lib/oxidized/nodes.rb:128:in synchronize' /home/oxidized/.gem/ruby/gems/oxidized-0.24.0/lib/oxidized/nodes.rb:128:inwith_lock’
/home/oxidized/.gem/ruby/gems/oxidized-0.24.0/lib/oxidized/nodes.rb:10:in load' /home/oxidized/.gem/ruby/gems/oxidized-0.24.0/lib/oxidized/nodes.rb:123:ininitialize’
/home/oxidized/.gem/ruby/gems/oxidized-0.24.0/lib/oxidized/core.rb:14:in new' /home/oxidized/.gem/ruby/gems/oxidized-0.24.0/lib/oxidized/core.rb:14:ininitialize’
/home/oxidized/.gem/ruby/gems/oxidized-0.24.0/lib/oxidized/core.rb:4:in new' /home/oxidized/.gem/ruby/gems/oxidized-0.24.0/lib/oxidized/core.rb:4:innew’
/home/oxidized/.gem/ruby/gems/oxidized-0.24.0/lib/oxidized/cli.rb:12:in run' /home/oxidized/.gem/ruby/gems/oxidized-0.24.0/bin/oxidized:8:in<top (required)>’
/usr/local/bin/oxidized:23:in load' /usr/local/bin/oxidized:23:in

This is similar to the error I got when validating the config file, so I edited the config file and added the line that the validate said I needed, right before line 11:

Yaml: PARSE_CUSTOM_TAGS

This did not fix the problem, but I think I’ll pursue this a little further before reinstalling.

Personally I’d recommend you install oxidized on a dev box so you understand how it works, and then from that you’ll be able to fix your prod box.

Have a look at the oxidized github page. It has quite extensive docs. Secondly there’s a few good guides on google for installing Oxidized.

My main tip would be instead of running oxidized with systemctl, start the daemon manually with verbose mode configured. It should give more debug for why it’s failing.

I no longer use oxidized (in a different job), but feel free to update here how you get on and I’ll see if I can help.

Thanks, that seems like a good suggenstion. I have poked around some on the server it’s supposedly installed on, and running gem list does not show that oxidized is installed. Running gem install oxidized, returns the error

ERROR: Error installing oxidized
net-ssh requires Ruby version >= 2.2.6.

When I run yum list ruby, it shows

Installed Packages
ruby.x86_64 2.0.0.648-35.el7,6

and does not show any available packages. I tried removing ruby and reinstalling, but got the same version. This is running on a VMware guest, so I’m going to go back to the snapshot I took this morning before messing with ruby, and setup a clean guest to try the install like you suggest.

you need to yum update

you have ruby 2.0.0 and need 2.2.6 or higher

or google how to update ruby on centos

I have built a new CentOS7 guest and installed the packages per the oxidized github page, down to the Running with Docker section. I don’t have any of that stuff installed at this point, no docker or git command on my server, and the instructions did not say anything about installing them. yum list docker does show it’s available to install, but the version (2:1.13.1-96.gitb2f74b2,el7.centos) is nothing like the minimum required version listed in the installation instrutctions (17.05.0-ce or higher). Do I need to find that version of docker? Do I even need to run with docker or is this optional?

Docker is completely optional.

The docs are written in sections, not as a list of options you must complete.

Since you’re centos, simply refer just to https://github.com/ytti/oxidized/blob/master/README.md#centos-oracle-linux-red-hat-linux

After that move to the configuration section.