Struggling with Librenms-service inside a Truenas 13.3 Jail

Help
1

First of all I have to thank @dvl for dedicating some of his time to my issue (I was the guy who bothered on your blog comment session; I’ll try to be more precise there as this comment can became a tad long :wink:

I ask to your questione about what I meant with “librenms user shell”, I meant lauching the commands manually after executing a “su - librenms”

I will explain what I did before on a VM (to test the installation process on FreeBSD) and on the TN Jail

############################## ON VBOX #################################

I experimented at first with LibreNMS on a Vbox VM (Freebsd 13.3-RELEASE guest) and things (after a lot of time spent pulling my last hairs) went good (by that I mean the librenms service started fine).

That is the VM validate output:

===========================================
Component | Version
--------- | -------
LibreNMS  | 24.12.0 (2024-12-18T00:55:13+01:00)
DB Schema | 2024_11_22_135845_alert_log_refactor_indexes (310)
PHP       | 8.3.15
Python    | 3.11.11
Database  | MySQL 8.0.39
RRDTool   | 1.9.0
SNMP      | 5.9.4.pre2
===========================================

[OK]    Installed from package; no Composer required
[OK]    Database connection successful
[OK]    Database connection successful
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    Database and column collations are correct
[OK]    Database schema correct
[OK]    MySQL and PHP time match
[OK]    Active pollers found
[OK]    Dispatcher Service is enabled
[OK]    Locks are functional
[OK]    Python wrapper cron entry is not present
[OK]    Redis is unavailable
[OK]    rrdtool version ok
[OK]    Connected to rrdcached
[FAIL]  Scheduler is not running
        [FIX]:
        sudo sh -c 'sed "s#/opt/librenms#/usr/local/www/librenms#" /usr/local/www/librenms/dist/librenms-scheduler.cron > /etc/cron.d/librenms-scheduler.cron'

and if I issue a “ps auxww | grep librenms” :

librenms 22175  0.0  2.8  101568  58448  -  I    14:57     0:01.16 php-fpm: pool librenms (php-fpm)
librenms 22244  0.0  2.8   99520  58088  -  I    14:57     0:02.13 php-fpm: pool librenms (php-fpm)
librenms 22268  0.0  2.9  101568  59260  -  I    14:57     0:02.53 php-fpm: pool librenms (php-fpm)
librenms 23147  0.0  0.1   12808   2348  -  Ss   15:27     0:00.01 daemon: /usr/local/bin/python3.11[23148] (daemon)
librenms 23148  0.0  1.6   62996  33824  -  S    15:27     0:00.99 /usr/local/bin/python3.11 /usr/local/www/librenms/librenms-service.py
root     23699  0.0  0.0     432    256  0  R+   15:42     0:00.00 grep librenms

and “ps auxww | grep www” :

www      18092   0.0  1.7   75668  34420  -  I    13:19     0:00.05 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      18149   0.0  1.7   75668  34420  -  I    13:19     0:00.04 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      21992   0.0  1.7   75668  34420  -  S    14:55     0:00.04 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      22034   0.0  1.7   75668  34420  -  I    14:55     0:00.03 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      22035   0.0  1.7   75668  34420  -  I    14:55     0:00.05 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      22036   0.0  1.7   75668  34420  -  I    14:55     0:00.04 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      22355   0.0  1.7   75668  34420  -  I    14:59     0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      22363   0.0  1.7   75668  34420  -  I    14:59     0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      22382   0.0  1.7   75668  34420  -  I    15:00     0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      22754   0.0  1.7   75668  34420  -  I    15:14     0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT

and “grep librenms /etc/rc.conf”:

librenms_enable="YES"
librenms_user="librenms"
librenms_group="librenms"

I have to say that there was some fiddlings with ACLs as per the docs https://docs.librenms.org/Installation/Install-LibreNMS/#set-permissions so let me describe what I think can be related :

“getfacl /var/run/librenms/”

# file: /var/run/librenms/
# owner: www
# group: www
         group:www:rwxpDdaARWc--s:fd-----:allow
    group:librenms:rwxpDdaARWc--s:fd-----:allow
            owner@:rwxp--aARWcCos:-------:allow
            group@:rwxp--a-R-c--s:-------:allow
         everyone@:r-x---a-R-c--s:-------:allow

“getfacl /var/run/librenms/librenms.pid”

# file: /var/run/librenms/librenms.pid
# owner: librenms
# group: www
         group:www:rwxpDdaARWc--s:------I:allow
    group:librenms:rwxpDdaARWc--s:------I:allow
            owner@:rw-p--aARWcCos:-------:allow
            group@:------a-R-c--s:-------:allow
         everyone@:------a-R-c--s:-------:allow

“getfacl /var/db/librenms/”

# file: /var/db/librenms/
# owner: www
# group: www
    group:librenms:rwxpDdaARWc--s:fd-----:allow
            owner@:rwxp--aARWcCos:-------:allow
            group@:rwxp--a-R-c--s:-------:allow
         everyone@:r-x---a-R-c--s:-------:allow

getfacl /usr/local/www/librenms/storage/

"# file: /usr/local/www/librenms/storage/"
# owner: www
# group: www
         group:www:rwxpDdaARWc--s:fd-----:allow
    group:librenms:rwxpDdaARWc--s:fd-----:allow
            owner@:rwxp--aARWcCos:-------:allow
            group@:rwxp--a-R-c--s:-------:allow
         everyone@:r-x---a-R-c--s:-------:allow

“getfacl /usr/local/www/librenms/bootstrap/cache/”

# file: /usr/local/www/librenms/bootstrap/cache/
# owner: librenms
# group: librenms
         group:www:rwxpDdaARWc--s:fd-----:allow
            owner@:rwxp--aARWcCos:-------:allow
            group@:rwxp--a-R-c--s:-------:allow
         everyone@:r-x---a-R-c--s:-------:allow

with this settings the librenms service says it’s running and the /var/run/librenms/librenms.pid file it’s created and contains the PID of the process.

############################## ON TRUENAS ###################################

Then (given the success) I started working in a Jail (basejail) on a Truenas 13.3-RELEASE host , in the meantime I edited freebsd.conf to use “latest” train and now that’s my validate.php output:

===========================================
Component | Version
--------- | -------
LibreNMS  | 25.1.0 (2025-01-17T06:28:40+01:00)
DB Schema | 2024_11_22_135845_alert_log_refactor_indexes (310)
PHP       | 8.3.16
Python    | 3.11.11
Database  | MySQL 8.0.39
RRDTool   | 1.9.0
SNMP      | 5.9.4.pre2
===========================================

[OK]    Installed from package; no Composer required
[OK]    Database connection successful
[OK]    Database connection successful
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    Database and column collations are correct
[OK]    Database schema correct
[OK]    MySQL and PHP time match
[OK]    Active pollers found
[OK]    Dispatcher Service is enabled
[OK]    Locks are functional
[OK]    Python wrapper cron entry is not present
[OK]    Redis is unavailable
[OK]    rrd_dir is writable
[OK]    rrdtool version ok

and “grep librenms /etc/rc.conf”:

librenms_enable="YES"
librenms_user="librenms"
librenms_group="librenms"

and if I issue a “ps auxww | grep librenms” :

librenms 57024  0.0  0.2   93640  57968  -  IJ   13:41   0:00.86 php-fpm: pool librenms (php-fpm)
librenms 57059  0.0  0.2   87496  51804  -  IJ   13:41   0:00.31 php-fpm: pool librenms (php-fpm)
librenms 61104  0.0  0.1   81352  45368  -  IJ   15:33   0:00.14 php-fpm: pool librenms (php-fpm)
librenms 56424  0.0  0.1  120824  40360 23- SJ   13:35   0:05.73 /usr/local/bin/python3.11 /usr/local/www/librenms/librenms-service.py

and “ps auxww | grep www” :

www      55333  0.0  0.1   66776  32836  -  IJ   13:09   0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      55334  0.0  0.1   66776  32836  -  IJ   13:09   0:00.02 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      55335  0.0  0.1   69848  32860  -  IJ   13:09   0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      55336  0.0  0.1   69848  32860  -  IJ   13:09   0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      55337  0.0  0.1   66776  32832  -  IJ   13:09   0:00.02 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      55953  0.0  0.1   69848  32860  -  IJ   13:24   0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      55954  0.0  0.1   66776  32836  -  IJ   13:24   0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      56839  0.0  0.1   69848  32832  -  IJ   13:38   0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      56957  0.0  0.1   66776  32784  -  SJ   13:41   0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      57052  0.0  0.1   66776  32804  -  IJ   13:41   0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT

“getfacl /var/run/librenms/”:

# file: /var/run/librenms/
# owner: www
# group: www
         group:www:rwxpDdaARWc--s:fd-----:allow
    group:librenms:rwxpDdaARWc--s:fd-----:allow
            owner@:rwxp--aARWcCos:-------:allow
            group@:rwxp--a-R-c--s:-------:allow
         everyone@:r-x---a-R-c--s:-------:allow

“getfacl /var/run/librenms/librenms.pid”

# file: /var/run/librenms/librenms.pid
# owner: librenms
# group: librenms
         group:www:rwxpDdaARWc--s:-------:allow
    group:librenms:rwxpDdaARWc--s:-------:allow
            owner@:rw-p--aARWcCos:-------:allow
            group@:------a-R-c--s:-------:allow
         everyone@:------a-R-c--s:-------:allow

“getfacl /var/db/librenms/”

# file: /var/db/librenms/
# owner: www
# group: www
    group:librenms:rwxpDdaARWc--s:fd-----:allow
            owner@:rwxp--aARWcCos:-------:allow
            group@:rwxp--a-R-c--s:-------:allow
         everyone@:r-x---a-R-c--s:-------:allow

getfacl /usr/local/www/librenms/storage/

# file: /usr/local/www/librenms/storage/
# owner: www
# group: www
    group:librenms:rwxpDdaARWc--s:fd-----:allow
            owner@:rwxp--aARWcCos:-------:allow
            group@:rwxp--a-R-c--s:-------:allow
         everyone@:r-x---a-R-c--s:-------:allow

“getfacl /usr/local/www/librenms/bootstrap/cache/”

# file: /usr/local/www/librenms/bootstrap/cache/
# owner: librenms
# group: librenms
    group:librenms:rwxpDdaARWc--s:fd-----:allow
            owner@:rwxp--aARWcCos:-------:allow
            group@:rwxp--a-R-c--s:-------:allow
         everyone@:r-x---a-R-c--s:-------:allow

Even upgrading to latest port (25.1.0) doesn’t change the outcome: pidfile created but empty…

At the end that is how I managed to have the librenms-service (dispatcher) running:

after getting in the librenms shell (su - librenms) I gave the command
nohup /usr/local/bin/python3.11 /usr/local/www/librenms/librenms-service.py &
and called it a day but that workaround makes me cringe… :cold_sweat:

Hope it makes sense and someone in this community will be able to nudge me in the right path, in case you need other details let me know.

I really cannot grasp why if I launch the librenms-service.py “by hand” it works and it doesn’t work if I use the service command (daemon could be bad behaving?!) instead

Regards

2

I am not sure of the implications of running the service not as www:

librenms_user=“librenms”
librenms_group=“librenms”

3

I did this change because of problems with permissions; if I leave the default I get

daemon: open: Permission denied
/usr/local/etc/rc.d/librenms: WARNING: failed to start librenms

Modified ACLs of log files

setfacl -R -m g:www:modify_set:fd:allow /var/log/librenms/librenms-service.log
setfacl -R -m g:www:modify_set:fd:allow /var/log/librenms/librenms.log

I don’t get the permission anymore error but there’s no improvement whatsoever…

4

first line of the rc.d script, append ’ -x’, then service librenms start will have lots of debugging output for you.

5

been there done that -x debug script output

it must be me but I cannot see anything clearly wrong…

6

Seems normal. Did it start? Or are we back to the original problem?

Anything in /var/log/librenms/?

7

I’m stuck with the original problem: service says it starts but it does not, if I issue service librenms status out responsabile that the service is not running, ps show no librenms-service.py process and the pidfile get created but it’s empty.

I will update with the logs as soon as I could get back home and reach for my pc.

Thank you as always

8

Back in front of my desk, on the Truenas Jail this is the content of the /var/log/librenms path

total 97
243912 -rw-r--r--  1 www       www         84K Feb 23 19:30 librenms.log
 15717 drwxr-xr-x  5 root      wheel       40B Feb 23 19:00 ..
243693 -rw-r--r--  1 www       www        194B Feb 23 01:08 maintenance.log
206266 -rw-r--r--  1 librenms  librenms    14K Feb 23 01:00 daily.log
201208 drwxrwxr-x  2 www       www         10B Feb 23 01:00 .
242921 -rw-r--r--  1 www       www        121K Feb 23 00:00 librenms.log.0
207239 -rw-rwxr--+ 1 www       www         47K Feb 22 00:00 librenms.log.1
207119 -rw-r--r--  1 librenms  librenms   5.9K Feb 21 12:51 poller_wrapper.log
207550 -rw-------+ 1 librenms  librenms     0B Feb 21 10:17 librenms-service.log
204750 -rw-r--r--  1 www       www        1.2M Feb 21 00:00 librenms.log.2

and that’s what’s inside librenms.log

[2025-02-23T19:09:00][ERROR] file_put_contents(/usr/local/www/librenms/storage/framework/cache/data/27/45/2745da5ffa1c30968efa55bffb4f58b2d7a690a8): Failed to open stream: Permission denied {"exception":"[object] (ErrorException(code: 0): file_put_contents(/usr/local/www/librenms/storage/framework/cache/data/27/45/2745da5ffa1c30968efa55bffb4f58b2d7a690a8): Failed to open stream: Permission denied at /usr/local/www/librenms/vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:204)"} 
[2025-02-23T19:14:00][ERROR] file_put_contents(/usr/local/www/librenms/storage/framework/cache/data/27/45/2745da5ffa1c30968efa55bffb4f58b2d7a690a8): Failed to open stream: Permission denied {"exception":"[object] (ErrorException(code: 0): file_put_contents(/usr/local/www/librenms/storage/framework/cache/data/27/45/2745da5ffa1c30968efa55bffb4f58b2d7a690a8): Failed to open stream: Permission denied at /usr/local/www/librenms/vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:204)"} 
[2025-02-23T19:19:00][ERROR] file_put_contents(/usr/local/www/librenms/storage/framework/cache/data/27/45/2745da5ffa1c30968efa55bffb4f58b2d7a690a8): Failed to open stream: Permission denied {"exception":"[object] (ErrorException(code: 0): file_put_contents(/usr/local/www/librenms/storage/framework/cache/data/27/45/2745da5ffa1c30968efa55bffb4f58b2d7a690a8): Failed to open stream: Permission denied at /usr/local/www/librenms/vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:204)"} 
[2025-02-23T19:24:00][ERROR] file_put_contents(/usr/local/www/librenms/storage/framework/cache/data/27/45/2745da5ffa1c30968efa55bffb4f58b2d7a690a8): Failed to open stream: Permission denied {"exception":"[object] (ErrorException(code: 0): file_put_contents(/usr/local/www/librenms/storage/framework/cache/data/27/45/2745da5ffa1c30968efa55bffb4f58b2d7a690a8): Failed to open stream: Permission denied at /usr/local/www/librenms/vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:204)"} 
[2025-02-23T19:30:00][ERROR] file_put_contents(/usr/local/www/librenms/storage/framework/cache/data/27/45/2745da5ffa1c30968efa55bffb4f58b2d7a690a8): Failed to open stream: Permission denied {"exception":"[object] (ErrorException(code: 0): file_put_contents(/usr/local/www/librenms/storage/framework/cache/data/27/45/2745da5ffa1c30968efa55bffb4f58b2d7a690a8): Failed to open stream: Permission denied at /usr/local/www/librenms/vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:204)"}

repeated 172 times… every 5 minutes, but at line 204 there’s this function

    public function put($path, $contents, $lock = false)
    {
        return file_put_contents($path, $contents, $lock ? LOCK_EX : 0);
    }

that I’m not able to understand why it cannot “open stream” since this are the permissions (/usa/local/www/librenms/storage it’s a symlink to /var/db/librenms/storage

ls -hilta /var/db/librenms/
total 10
201205 drwxrwxr-x+  6 www   www          6B Feb 21 13:07 storage
 15728 drwxr-xr-x  19 root  wheel       22B Feb 21 13:07 ..
205824 drwxrwxr-x+  4 www   librenms     4B Feb 20 19:47 rrd
201204 drwxrwxr-x+  4 www   www          4B Feb 20 18:32 .

getfacl /var/db/librenms/storage
# file: /var/db/librenms/storage
# owner: www
# group: www
    group:librenms:rwxpDdaARWc--s:fd-----:allow
            owner@:rwxp--aARWcCos:-------:allow
            group@:rwxp--a-R-c--s:-------:allow
         everyone@:r-x---a-R-c--s:-------:allow

:nauseated_face:

9

I finally found it!!

the cause of the problem with the non-starting librenms service was the daemon executable in my basejail on TN 13.3!
I stumbled on this post while scavenging for a solution damn daemon

so I copied the host daemon binari in /usr/local/bin (only writable path in $PATH) and vuallà; the service started without a fuss

Thank you @dvl for giving me support on that issue; maybe it will help someone in the same situation, it’s not a really common setup having a 13.4 basejail on a 13.3 TN host but you never know!

See ya!

10

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.