Crash oxidized after upgrade to Debian 12

julianmaciaslopez · 26 May 2025 11:35

Hello,

My installation was working fine until I had to upgrade from Debian11 to Debian12 due to client security restrictions.

Attached are the configuration files:

Config oxidized file:

root@librenms-com:~# cat /home/oxidized/.config/oxidized/config 
---
username: test
password: test
model: junos
resolve_dns: true
interval: 28800
use_syslog: true
debug: false
threads: 30
timeout: 20
retries: 2
prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/
rest: librenms-com.X.es:8888
next_adds_job: true
vars: {}
groups: {}
models: {}
pid: "/home/oxidized/.config/oxidized/pid"
crash:
  directory: "/home/oxidized/.config/oxidized/crashes"
  hostnames: false
stats:
  history_size: 10
input:
  default: telnet, ssh
  debug: true
  ssh:
    secure: false
  http:
    ssl_verify: true
  ftp:
    passive: true
  utf8_encoded: true
output:
  default: git
  git:
    user: Oxidized
    email: [email protected]
    repo: "/home/oxidized/configs/git-repos/default.git"
source:
  default: http
  debug: true
  http:
    url: https://librenms-com.X.es/api/v0/oxidized
    scheme: https
    secure: true
    map:
      name: hostname
      model: os
      group: group
    headers:
      X-Auth-Token: X
model_map:
  juniper: junos
  cisco: ios

config nginx file:

root@librenms-com:~# cat /etc/nginx/sites-enabled/librenms.vhost
server {
listen 80;
listen [::]:80;
server_name librenms-com.X.es;
return 301 https://$server_name$request_uri;
}

server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name librenms-com.X.es;
root /opt/librenms/html;
index index.php;


ssl_certificate "/etc/certificados/fullchain.pem";
ssl_certificate_key "/etc/certificados/privkey.pem";
ssl_session_cache shared:SSL:1m;
ssl_session_timeout  10m;

access_log /opt/librenms/logs/access_log;
error_log /opt/librenms/logs/error_log;

charset utf-8;
 gzip on;
 gzip_types text/css application/javascript text/javascript application/x-javascript image/svg+xml text/plain text/xsd text/xsl text/xml image/x-icon;
 location / {
  try_files $uri $uri/ /index.php?$query_string;
 }
 location /api/v0 {
  try_files $uri $uri/ /api_v0.php?$query_string;
 }
 location ~ [^/]\.php(/|$) {
  fastcgi_pass unix:/run/php-fpm-librenms.sock;
  fastcgi_split_path_info ^(.+\.php)(/.+)$;
  include fastcgi.conf;
 }
 location ~ /\.(?!well-known).* {
  deny all;
 }
}

Output ./validate.php:

root@librenms-com:~# su - librenms 
librenms@librenms-com:~$ ./validate.php 
===========================================
Component | Version
--------- | -------
LibreNMS  | 25.5.0-45-gc4e31acd4 (2025-05-25T17:24:35+02:00)
DB Schema | 2025_05_20_084533_dashboard_admin_move_full_shared (340)
PHP       | 8.4.7
Python    | 3.11.2
Database  | MariaDB 10.11.11-MariaDB-0+deb12u1
RRDTool   | 1.7.2
SNMP      | 5.9.3
===========================================

[OK]    Composer Version: 2.8.9
[OK]    Dependencies up-to-date.
[OK]    Database Connected
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    Database and column collations are correct
[OK]    Database schema correct
[OK]    MySQL and PHP time match
[OK]    Distributed Polling setting is enabled globally
[OK]    Connected to rrdcached
[OK]    Active pollers found
[OK]    Dispatcher Service is enabled
[OK]    Locks are functional
[OK]    No active python wrapper pollers found
[OK]    Redis is functional
[FAIL]  Python3 module issue found: 'Required packages: ['PyMySQL!=1.0.0', 'python-dotenv', 'redis>=4.0', 'setuptools', 'psutil>=5.6.0', 'command_runner>=1.3.0']
Package not found: The 'command_runner>=1.3.0' distribution was not found and is required by the application
'
	[FIX]: 
	pip3 install -r /opt/librenms/requirements.txt
[OK]    rrdtool version ok
[OK]    Connected to rrdcached

Config Oxidized in LibreNMS:

image1348×376 16.9 KB
If I ask which teams belong to the polling list, it returns these correctly:

librenms@librenms-com:~$ curl -k -X GET -H 'X-Auth-Token:X' -i https://librenms-com.X.es/api/v0/oxidized
HTTP/2 200 
server: nginx/1.22.1
content-type: application/json
cache-control: no-cache, private
date: Mon, 26 May 2025 11:21:51 GMT
access-control-allow-origin: *
[
    {
        "hostname": "10.X.6.2",
        "os": "junos",
        "ip": "10.X.6.2"
    },
    {
        "hostname": "10.X.6.3",
        "os": "junos",
        "ip": "10.X.6.3"
    },
...

The service is running and extracting configurations:

root@librenms-com:~# systemctl status oxidized.service 
● oxidized.service - Oxidized - Network Device Configuration Backup Tool
     Loaded: loaded (/etc/systemd/system/oxidized.service; enabled; preset: enabled)
     Active: active (running) since Mon 2025-05-26 12:13:30 CEST; 1h 13min ago
   Main PID: 1277 (oxidized)
      Tasks: 8 (limit: 77079)
     Memory: 155.6M
        CPU: 2min 18.440s
     CGroup: /system.slice/oxidized.service
             └─1277 "puma 6.6.0 (tcp://librenms-com.X.es:8888) [/]"

may 26 13:26:49 librenms-com oxidized[1277]: lib/oxidized/worker.rb: 1 jobs running in parallel
may 26 13:26:50 librenms-com oxidized[1277]: lib/oxidized/worker.rb: 1 jobs running in parallel
may 26 13:26:51 librenms-com oxidized[1277]: lib/oxidized/worker.rb: 1 jobs running in parallel
may 26 13:26:52 librenms-com oxidized[1277]: lib/oxidized/worker.rb: 1 jobs running in parallel
may 26 13:26:53 librenms-com oxidized[1277]: lib/oxidized/worker.rb: 1 jobs running in parallel
may 26 13:26:54 librenms-com oxidized[1277]: lib/oxidized/worker.rb: 1 jobs running in parallel
may 26 13:26:55 librenms-com oxidized[1277]: lib/oxidized/worker.rb: 1 jobs running in parallel
may 26 13:26:56 librenms-com oxidized[1277]: lib/oxidized/worker.rb: 1 jobs running in parallel
may 26 13:26:57 librenms-com oxidized[1277]: lib/oxidized/worker.rb: 1 jobs running in parallel
may 26 13:26:58 librenms-com oxidized[1277]: lib/oxidized/worker.rb: 1 jobs running in parallel

But if I do a curl there seems to be a problem with the certificates or the handshake::

root@librenms-com:~# curl -v -f https://librenms-com.X.es:8888
*   Trying 172.X.X.13:8888...
* Connected to librenms-com.X.es (172.X.X.13) port 8888 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* OpenSSL/3.0.16: error:0A00010B:SSL routines::wrong version number
* Closing connection 0
curl: (35) OpenSSL/3.0.16: error:0A00010B:SSL routines::wrong version number

And the list of devices does not appear on the web, as it did before the update:

image1887×449 18 KB
The configuration is not even visible on the equipment:

image1898×353 28 KB

If anyone has any ideas and can help, I would greatly appreciate it.

rhinoau · 27 May 2025 02:18

I don’t use oxidized any long so unsure on that front - however this error is usually when you access a HTTP resource with https:// URI.

What do you get using just http:// in that request?

julianmaciaslopez · 27 May 2025 07:04

Hello,

First of all, thank you for your reply. I understand that it is a https/http/certified/encryption problem, but as the error was not occurring before the upgrade, I thought: maybe some package or fix not installed.

This is the output without https:

root@librenms-com:~# curl -v -f http://librenms-com.X.es:8888
*   Trying 172.X.X.13:8888...
* Connected to librenms-com.X.es (172.X.X.13) port 8888 (#0)
> GET / HTTP/1.1
> Host: librenms-com.X.es:8888
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 403 Forbidden
< content-type: text/plain
< x-content-type-options: nosniff
< Content-Length: 18
* The requested URL returned error: 403
* Closing connection 0
curl: (22) The requested URL returned error: 403

Roger · 27 May 2025 07:13

Your git user is written with a capital O but the path suggests the user is oxidized.

Not sure if that is a problem or not and I don’t want to risk crashing my own oxidized-repo to test but maybe try changing it to lower-case and see if it helps?

julianmaciaslopez · 27 May 2025 07:22

Thank you for your reply. I have changed it and restarted the service and nothing changes.

Roger · 27 May 2025 08:15

Weird that you’re getting a 403.
Maybe you’re running into something similar to this thread:

julianmaciaslopez · 27 May 2025 09:32

Wow thank you so much.

I added in the line: set :host_authorization, { permitted_hosts: [] } in the file /var/lib/gems/3.1.0/gems/oxidized-web-0.15.1/lib/oxidized/web/web/webapp.rb and it works:

root@librenms-com:/var/lib/gems/3.1.0/gems/oxidized-web-0.15.1/lib/oxidized/web# curl -v -f http://librenms-com.X.es:8888
*   Trying 172.29.128.13:8888...
* Connected to librenms-com.X.es (172.X.X.13) port 8888 (#0)
> GET / HTTP/1.1
> Host: librenms-com.X.es:8888
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 302 Found
< location: http://librenms-com.X.es:8888/nodes
< content-type: text/html;charset=utf-8
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< Content-Length: 0
< 
* Connection #0 to host librenms-com.X.es left intact

and is now available on LibreNMS.

devhubpl · 27 May 2025 22:12

Hey.
Maybe you should update the gems oxidized and oxidized-web to the latest versions. Look at the changelog 0.15.1->0.16.0 In the web.rb file WebApp.set :host_authorization, { permitted_hosts: vhosts } was added
You can try:

gem install oxidized
gem install oxidized-web oxidized-script

or

gem update

julianmaciaslopez · 28 May 2025 07:18

I have updated gem and it is also working. Thanks for the contribution.

system · 4 June 2025 07:18

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.