Oxidized public key auth no longer working

Hi Everyone,

I’ve noticed that Oxidized can no longer access my devices using public key authentication. Best I can tell this started happening late last year. It had been working fine for about two years previously. I’m currently running Oxidized version 0.28.0.

Oxidized still appears to connect occasionally, although the “Sync Status” for most devices says “no connection”, the “Last Sync” stat is still updating regularly. I can see the previously pulled configs that are stored in my git repo in the LibreNMS GUI, but no new configs are being stored. When I check debug.log I see the following error when Oxidized is attempting to authenticate to devices:

E, [2021-06-08T13:16:15.460137 #1770] ERROR – : 172.20.20.1 raised NoMethodError with msg “undefined method `each’ for “publickey”:String”, 172.20.20.1 saved

I’ve done my best Googlefu but haven’t found much information on what this error means, but it clearly doesn’t like something about using public key authentication. To be clear - ssh public key authentication to these devices works using the user and key set up for Oxidized and it works from the server that LibreNMS and Oxidized both live on, so it’s not an issue with the key itself or anything to do with the network path.

Here is a sanitized version of my Oxidized config.

username: user
password: password
model: enterasys
interval: 86400
log: ~/.config/oxy/logs/debug.log
use_syslog: false
debug: true
threads: 30
timeout: 300
retries: 3
prompt: !ruby/regexp /^([\[email protected]]+[#>]\s?)$/
rest: 127.0.0.1:8888
next_adds_job: false
vars: {}
groups:
  enterasys:
    username: user
    password: password
  hp:
    username: user
    vars:
      auth_methods: publickey
      ssh_keys: "home/oxy/.ssh/id_rsa"
  arubaos-cx:
    username: user
    vars:
      auth_mehtods: publickey
      ssh_keys: "home/oxy/.ssh/id_rsa"
  hp2910al:
    username: user
    password: password
  cisco:
    username: user
    password: password
    vars:
      enable: enablepass
  asa:
    username: user
    password: password
    vars:
      enable: enablepass
models: {}
pid: "/home/oxy/.config/oxidized/pid"
input:
  default: ssh
  debug: true
  ssh:
    secure: false
output:
  default: git
  git:
    user: oxy
    email: [email protected]
    single_repo: true
    repo: "/home/oxy/devices.git"
source:
  default: http
  debug: true
  http:
    url: https://127.0.0.1:6501/api/v0/oxidized
    scheme: https
    secure: false
    delimiter: !ruby/regexp /:/
    map:
      name: hostname
      model: os
      group: group
    headers:
      X-Auth-Token: 'API Token'
model_map:
  cisco: ios
  juniper: junos
  asa: asa
  enterasys: enterasys
  arubaos-cx: aoscx

Crash log from one of the switches failing to connect with public key authentication:

2021-06-09 16:42:33 UTC
undefined method `each' for "publickey":String [NoMethodError]
--------------------------------------------------
/var/lib/gems/2.3.0/gems/net-ssh-5.2.0/lib/net/ssh/authentication/session.rb:71:in `authenticate'
/var/lib/gems/2.3.0/gems/net-ssh-5.2.0/lib/net/ssh.rb:250:in `start'
/var/lib/gems/2.3.0/gems/oxidized-0.28.0/lib/oxidized/input/ssh.rb:27:in `connect'
/var/lib/gems/2.3.0/gems/oxidized-0.28.0/lib/oxidized/node.rb:67:in `run_input'
/var/lib/gems/2.3.0/gems/oxidized-0.28.0/lib/oxidized/node.rb:44:in `block in run'
/var/lib/gems/2.3.0/gems/oxidized-0.28.0/lib/oxidized/node.rb:38:in `each'
/var/lib/gems/2.3.0/gems/oxidized-0.28.0/lib/oxidized/node.rb:38:in `run'
/var/lib/gems/2.3.0/gems/oxidized-0.28.0/lib/oxidized/job.rb:9:in `block in initialize'

I’m happy to supply more information as needed if that will help. I appreciate any assistance.

I agree that it appears to be an issue with the parsing of your auth methods.

Looking at the Oxidized docs, they enclose their auth method in quotations like:
auth_methods: “publickey”

Do you have that as well?

Hi shenanigrins,

Thanks very much for the reply. I don’t currently have the auth method ‘publickey’ wrapped in double quotes within the config. I see that it is listed that way when an array of possible auth methods is used. It had been working previously without being enclosed in double quotes, but I’ll update my config and give it a try that way and report back. Thanks for the suggestion - I really appreciate it.

Unfortunately wrapping “publickey” in double quotes within the config did not resolve the issue. I also noticed a typo in my arubaos-cx group and fixed that, but that also had no effect.

Try removing this line;

I’ll give it a try. Thanks for the suggestion.

I tried your suggestion, and unfortunately the issue persists. I appreciate the assistance, though.

I’ve also just noticed that the few remaining switches that use password authentication are also failing to pull new configs, but they do appear to connect and authenticate despite throwing the following error:

E, [2021-06-10T06:51:52.147741 #2667] ERROR – net.ssh.authentication.agent[16493fc]: could not connect to ssh-agent: Agent not configured

I haven’t had time yet to look into this too deeply. I’ll update my main post once I’ve done a bit more troubleshooting. Thanks again.

Ok, it makes sense why I get the “could not connect to ssh-agent: Agent not configured” error in this instance - this switch does not have the ability to use a public key for auth like the others do, thus it fails and moves on to the password auth method, which succeeds. Yet Oxidized still fails to download new config changes.

@shenanigrins I got it working finally. You were pretty much correct in your suggestion, except that I needed to add the brackets as well to indicate an array of one item. So,

auth_methods: ["publickey"]

As I mentioned before, it had previously worked the way I had it originally so I didn’t expect it to require the new syntax.

The other issue I was having with the switches that do not permit publickey auth turned out to be a prompt issue. The switches in question are Enterasys models. I ended up having to change the prompt in the enterasys.rb file to

/^([\[email protected]()-]+->\s?)$/

Thanks again for your assistance.

Nice. Glad to hear it’s working!