Feeding oxidized from librenms devices (API) not working

Help Oxidized
1

Hi,
I’m an absolutly newbie to linux and this kind of stuff.
I’m working as an network technician but i never cared about IT systems.
But at my new work there is no one else, so i have to care for myself…

Earlier I was working with rancid and I guess webcvs. There was a nice web GUI for adding devices and viewing configurations and diffs and comparing choosable versions. I guess the web frontend was self programmed by a colleague, but I’m not sure. I didn’t find something like this to download and install as ready2go.

So thats why I want to try librenms, because I thought it would be easier 'cause of oxidized integration.

I use centos OVA. It was difficult to get oxidized started, and i don’t know how, but it is working (not as service/deamon). And after an “hack” git as output and librenms versioning works.

But my problem now is the other direction. I do not want to add devices manually in the router.db of oxidized. I want librenms to feed oxidized.

I did it like documentation from librenms: https://docs.librenms.org/Extensions/Oxidized/#feeding-oxidized

source:
  default: http
  debug: false
  http:
    url: https://127.0.0.1/api/v0/oxidized
    map:
      name: hostname
      model: os
      group: group
    headers:
      X-Auth-Token: '01582bf94c03104ecb7953dsadsadwed'

Then I get the error:
[librenms@localhost oxidized]$ oxidized
Failed to open TCP connection to 127.0.0.1:443 (Connection refused - connect(2) for “127.0.0.1” port 443)

If I switch to http

...
  http:
    url: https://127.0.0.1/api/v0/oxidized
...

Then I get the error message:
[librenms@localhost oxidized]$ oxidized
no implicit conversion of String into Integer

Any idea?

Hint: The OVA was imported in Virtualbox on by desktop, it is not a productive environment.
the only thing librenms should do is backup config with revisioning. Monitoring is done by PRTG (which does not have this feature).

2

Welcome to Linux. I’m relatively new to working with Linux myself but I figure you won’t learn it unless you do it and it’s an incredibly useful skill to have.

Looks to me like you followed this: https://docs.librenms.org/Extensions/Oxidized/ and those instructions kind of suck because it tells you to use https:// to connect to the API. Try changing it to http://

Then try running this from the command line:
curl -H “X-Auth-Token:YOUR_API_KEY” http://127.0.0.1/api/v0/oxidized

If you don’t get output, look at SELINUX and look at firewall.

3

Incredible fast.
Well as I mentioned earlier https and http do not work, but with different error message.

Here is the output of the curl command. I took the value for X-Auth-Token from the librenms guide.

[librenms@localhost oxidized]$ curl -H “X-Auth-Token:01582bf94c03104ecb7953dsadsadwed” http://127.0.0.1/api/v0/oxidized
{"message":"Unauthenticated."}[librenms@localhost oxidized]$
4

Okay Update.
I guess there was at least one mistake from me, taking the example Token?

Now I create an API Token Hash in the librenms WebGUI and took this one for authentication.

oxidized is now staring, but the API Test with the curl continues to fail.

@Killo_Richards:
“If you don’t get output, look at SELINUX and look at firewall.”
How to? I’m using the OVA-Image so I hope everything is preconfigured correctly. But I think because of the authentication error message, it is still an other problem…

5

Me personally, I created an oxidized user in LibreNMS and gave it global read and then I assigned the API key for the oxidized user. Take that key and run it into curl.

Also check the http:// https:// thing, firewall and SELINUX (I’m not sure what flavor of linux you’re running, I don’t have SELINUX but on other systems like Centos or RHEL SELINUX has given me fits).

EDIT–> Here’s what the walkthrough says to do for SELINUX

SELinux

If you’re runnng SELinux, you’ll need to allow httpd to connect outbound to the network, otherwise Oxidized integration in the web UI will silently fail:

setsebool -P httpd_can_network_connect 1

Edit again, are you running the Centos or the Ubuntu OVA?
on ubuntu you can run:

sudo ufw status

to see if a firewall is enabled.
on centos you can run:

sudo systemctl status firewalld
6

Cause I did not get oxidized on ubuntu OVA to work, actually I am Using the centos based OVA Image.

https --> connection refused
http --> curl error message as above

[librenms@localhost oxidized]$ sudo systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-08-12 12:50:27 UTC; 34s left
     Docs: man:firewalld(1)
 Main PID: 420 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─420 /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopid

Aug 12 12:50:26 localhost.localdomain systemd[1]: Starting firewalld - dynami...
Aug 12 12:50:27 localhost.localdomain systemd[1]: Started firewalld - dynamic...
Aug 12 12:50:28 localhost.localdomain firewalld[420]: WARNING: AllowZoneDrift...
Hint: Some lines were ellipsized, use -l to show in full.
[librenms@localhost oxidized]$
7

I honestly don’t know if firewalld would stop localsystem from accessing itself. FOR TROUBLESHOOTING PURPOSES ONLY you should be able to do

sudo systemctl stop firewalld

and try the curl command again.

8

I have stopped firewall daemon, but no difference

[librenms@localhost oxidized]$ sudo curl -H “X-Auth-Token:19e35a0b60510fc81852209357b8df0c” http://127.0.0.1/api/v0/oxidized {"message":"Unauthenticated."}[librenms@localhost oxidized]$

Even firewall daemon is disabled:
[librenms@localhost ~]$ sudo curl -H “X-Auth-Token:19e35a0b60510fc81852209357b8df0c” https://127.0.0.1/api/v0/oxidized
curl: (7) Failed connect to 127.0.0.1:443; Connection refused

9

OK. Curl merror message is the one.
But I guess the funtion is working now.
I add an device to librenms and after a while the config is being captured.
I will test another one.

Then the problem was layer8, myself, with the wrong api key…

10

I struggled with this myself, I just can’t remember exactly what my issue was.
If I do:

root@librenms:~# curl -H “X-Auth-Token:my_token” https://127.0.0.1/api/v0/oxidized
curl: (7) Failed to connect to 127.0.0.1 port 443: Connection refused

if I do:
root@librenms:~# curl -H “X-Auth-Token:WRONG_TOKEN” http://127.0.0.1/api/v0/oxidized
{“message”:“Unauthenticated.”}root@librenms:~#

double check your token and http vs https. unauthenticated is a good sign…looks like bad token.

11

Hi Killo,

see post above. It works now, in spite of the curl message (unauthenticated)…

Solution was to create an User with Global Read and the create an API Key for this user and use this Token for API call.

Thanks for your help!

12

yeah I don’t think that’s in the instructions / how to. I just did it because it seemed to make sense from a security standpoint. Your curl command should work though. It sounds like whatever API key you have in your oxidized config is valid and the one you are using for your curl command is not valid (verify this) – because as you get further into configuring oxidized you’re going to want to run CURL to see what LibreNMS is feeding oxidized and it will facilitate setting up your groups.

13

[librenms@localhost /]$ curl -H “X-Auth-Token:19e35a0b60510fc81852209357b8df0c” http://localhost/api/v0/oxidized
{“message”:“Unauthenticated.”}[librenms@localhost /]$

url: http://localhost/api/v0/oxidized
headers:
 X-Auth-Token: '19e35a0b60510fc81852209357b8df0c'

19e35a0b60510fc81852209357b8df0c
19e35a0b60510fc81852209357b8df0c

It is absolutely the same Token Hash Value :frowning:

14

I’m genuinely perplexed. I did a couple of tests here and the only way I could re-create an “unauthenticated” is by either using the wrong API OR by having a leading space in the header data

curl -H “(SPACE HERE) X-Auth-Token:some_token”

I vaguely recall having a similar problem where it wasn’t working right and I was doing copy/paste of the API and I think maybe I had an issue in the spacing because I was just running arbitrary commands that I found online trying to find an answer. Forgive me if you already know this but from what little I know, curl is like a web browser without a gui. So when we do curl -H we are supplying header information with the request and the header information for this particular bit is a password of sorts.

You are going to want to figure this out because chances are you have multiple network devices that you want to backup and they have different OS’s and different usernames/passwords so LibreNMS is going to identify those but you’re not going to know what they are being identified as and you’re going to need to be able to curl to see what it’s doing so that you can tell LibreNMS to put them into different buckets and then create different buckets in Oxidized to receive them and connect / different credentials and different commands to connect for backups.

15

yeah, that problem will face me, too.
LAN switches --> credential 1
DMZ Switches --> credential 2
Firewalls --> individual credentials
etc.

There is still a lot of work.

Using rancid at my old employer was much easier.
Web frontend add IP, choose template, optionally description, click.
Device added to rancid and first configuration snapshot was done.
But as I do not know how it was realized (which OS and componentens) i can’t recreate it :frowning:

I thought I could use librenms as an easy webfrontend for oxidized… I was so wrong… :wink:

16

As I mentioned, I’m not much of a Linux guy. Awhile back I forced myself to learn more about it because frankly I’ve worked in IT for a long time and I noticed the demand for Linux guys is increasing and they tend to get paid more…So when I took a new job, I ended up standing up an instance of Observium. I looked at rancid initially but set it up to use oxidized…Oxidized in my opinion is superior to Rancid. I’m not even sure if rancid is being maintained anymore. If you had a web front-end on rancid, it might have been something custom, I don’t know.

With observium, which is very similar to LibreNMS, actually LibreNMS is a fork of observium – I had to manually add hosts into a router.db file. It worked, it wasn’t super pretty. I’m sure guys who know more could probably script something to dump the data from Observium and write this into a new config.

A few months back I abandoned observium after looking at LibreNMS…and it’s actually lightyears ahead of observium in my opinion and the integration, while still a work in progress is really quite nice and it was thought out.

I’d say to stick with LibreNMS + oxidized. It’s going to be worth it once you get it going. Yeah there’s the learning curve and it’s sometimes painful and frustrating as hell, trust me I get it…but just keep banging on it and you’ll appreciate that you’ve got an enterprise NMS system for nothing that does stuff that normally costs thousands upon thousands of dollars annually – plus you’re building your own skillset in the process.

17

On CentOS it looks like curl needs another syntax:

NOT: curl -H “X-Auth-Token:19e35a0b60510fc81852209357b8df0c” http://localhost/api/v0/oxidized

BUT: curl -H ‘X-Auth-Token: 19e35a0b60510fc81852209357b8df0c’ http://localhost/api/v0/oxidized

[librenms@localhost /]$ curl -H 'X-Auth-Token: 19e35a0b60510fc81852209357b8df0c' http://localhost/api/v0/oxidized
[
{
    "hostname": "localhost",
    "os": "linux",
    "ip": "172.19.32.227"
},
{
    "hostname": "it-switch-b221",
    "os": "procurve",
    "ip": "172.18.10.215"
},

And damn, it shows me, that the group information from librenms are missing :rage:

In librenms the groups are correct.
and oxidized config is

groups:
  DMZ:
    username: admin
    password: pass1
    model: procurve
  LAN:
    username: manager
    password: Pass2
    model: procurve
source:
  default: http
  debug: false
  http:
    url: http://localhost/api/v0/oxidized
    map:
      name: hostname
      model: os
      group: group
    headers:
     X-Auth-Token: '19e35a0b60510fc81852209357b8df0c'
1 Like
18

sorry for the delay…had a meltdown here at work. Catastrophe averted but we had to do the Sherlock Holmes thing. First off, nice job on figuring out the curl thing…Linux quirks. hahaha.

So some things about my environment that may be different from yours. We use IP almost exclusively for all of our network equipment, it’s not in DNS so it’s registered in librenms as IP only. When I run curl, I don’t data for “ip” but I do get a group and it throws it into default…not sure how or why, let me look at my librenms config.

$config[‘oxidized’][‘enabled’] = TRUE;
$config[‘oxidized’][‘url’] = ‘http://127.0.0.1:8888’;
$config[‘oxidized’][‘features’][‘versioning’] = true;
$config[‘oxidized’][‘reload_nodes’] = true;
$config[‘oxidized’][‘group_support’] = true;
$config[‘oxidized’][‘default_group’] = ‘default’;

That explains why I have a default group but doesn’t explain why I don’t have IP when I curl…So when Libre spits information into oxidized, it has that default group with default credentials that covers 99% of my equipment. The other 1% I handle through groups and that’s where overrides come in:

this document there is a section: called “Creating Overrides.”

I couldn’t quite make sense of what they were saying, other than you’re going to transform what oxidized has in the DB when it shoots to the API. For me I did this:

Oxidized host overrides

$config[‘oxidized’][‘maps’][‘group’][‘hostname’][] = array(‘regex’ => ‘/10.11.1.10/’, ‘group’ => ‘default’);
$config[‘oxidized’][‘maps’][‘group’][‘hostname’][] = array(‘regex’ => ‘/10.11.1.1/’, ‘group’ => ‘fortios’);
$config[‘oxidized’][‘maps’][‘group’][‘hostname’][] = array(‘regex’ => ‘/10.11.19.170/’, ‘group’ => ‘aosw’);

There’s probably a better way of doing this but I don’t really have much outside of these couple of “oddball” hosts. I figure that if you had locations with different passwords or something you could take an entire location and transform them to DMZ or whatever for example.

Next, (I think rancid is really similar in that it uses rb files that has the connection info for various devices. On my system the RB files are located in

./var/lib/gems/2.5.0/gems/oxidized-0.28.0/lib/oxidized/model/

Depending on what model your device is, you want to match. I just make this the “group” because it makes the most sense to me. Then in my oxidized config I have:

groups:
fortios:
username: myFortiAdmin
password: myFortiPassword
model: fortios

some_other_os:
username: my_other_os_username
password: my_other_password
model: some_other_os

Also IMPORTANT NOTE: I think I did find a potential bug in the API and the overrides. At one point I changed the first host that showed up in the api curl to something arbitrary and when I took out the override, the API continued to spit out the arbitrary group. I had to override it back to default…so just be aware of this before you do something globally and have 1000 devices that are getting overriden.

19

Thanks again!

I had expected that the settings in thelibre WebGUI for external oxidized should be enough. Your settings are the same I did in the WebGUI.
But I cant find the settings in the config.php of librenms…

When I enter this
$config[‘oxidized’][‘url’] = ‘http://127.0.0.1:8888’;

an error message occurs and librenms is not responding any longer
[librenms@localhost ~]$ oxidized
no implicit conversion of String into Integer

Same for
$config[‘oxidized’][‘maps’][‘group’][‘hostname’][] = array(‘regex’ => ‘/192/’, ‘group’ => ‘DMZ’);

If i remove these lines, librenms WebGUI is respnding again and oxidized can be started.

With only:
$config[‘oxidized’][‘enabled’] = TRUE;
$config[‘oxidized’][‘features’][‘versioning’] = true;
$config[‘oxidized’][‘group_support’] = true;
$config[‘oxidized’][‘default_group’] = ‘default’;
$config[‘oxidized’][‘reload_nodes’] = true;

everythings stays as before. No config backup for my dmz switch.

12h, i guess i should go home for today^^

20

@rgrossmann-ewr

Any additional configuration options for librenms will not be in the config.php until you add them.

probably obvious but anytime you tweak your config.php file and librenms stops working, something is wrong in the config. It’s not a horrible practice to make a backup of the file “sudo cp config.php config.bak” before you go in and start editing.

I’m unsure how the web-GUI interacts with what’s in the config. I can’t recall if it was observium or librenms where modifying the config.php would actually display in the web-GUI and lock the setting. I don’t think that the web-GUI actually writes to the config. If I had to guess, I would say that there’s probably some area of the database where the web-GUI settings are kept and, again just guessing that by having a value plugged into the GUI and then going directly into config.php and editing might blow stuff up.
I would try pulling the stuff out of the web-GUI and just writing directly to the config and see what happens.

Also after making changes to config.php be sure to sudo systemctl stop [apache/nginx/httpd] && systemctl start apache/nginx/httpd] then run your curl again and see what happens with the overrides. It took me some trial and error to get it working right – but you’re definitely on the right track here.

A neat feature of librenms/oxidized is that you can have librenms note when a configuration change has been made to a device and it will spawn an oxidized backup. I can’t recall if this is done in both the config.php and the oxidized config or not. One problem I ran into with this is my firewall changes keys randomly in the config and this was causing multiple backups to run every hour…I ended up just disabling oxidized backups for this - which was humorous because I went through all of the crap to get the groups working and separating out my FW’s and such and then I ultimately just disabled it.