As there aren’t any specific installation instructions for rrdcached on Centos 8 I followed those available for Centos 7. I created the /etc/systemd/system/rrdcached.service as specified but when I try to start the service I get the following error:
[root@localhost librenms]# systemctl enable --now rrdcached.service
Job for rrdcached.service failed because the control process exited with error code.
See "systemctl status rrdcached.service" and "journalctl -xe" for details.
systemctl status gives the following detail:
[root@localhost librenms]# systemctl status rrdcached.service
● rrdcached.service - Data caching daemon for rrdtool
Loaded: loaded (/etc/systemd/system/rrdcached.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2020-07-28 11:29:03 BST; 3s ago
Process: 2357 ExecStart=/usr/bin/rrdcached -w 1800 -z 1800 -f 3600 -s librenms -U librenms -G librenms -B -R -j /var/tmp -l unix:/run/rrdcached.sock -t 4 -F -b /opt/librenms/rrd/ (code=exited, status=3)
Jul 28 11:29:03 localhost.localdomain systemd[1]: Starting Data caching daemon for rrdtool...
Jul 28 11:29:03 localhost.localdomain rrdcached[2357]: Failed to create base directory '/opt/librenms/rrd/': Permission denied
Jul 28 11:29:03 localhost.localdomain systemd[1]: rrdcached.service: Control process exited, code=exited status=3
Jul 28 11:29:03 localhost.localdomain systemd[1]: rrdcached.service: Failed with result 'exit-code'.
Jul 28 11:29:03 localhost.localdomain systemd[1]: Failed to start Data caching daemon for rrdtool.
This is an error that doesn’t make a lot of sense to me as rrddcached is set to run as the librenms user in the librenms group and the /opt/librenms/rrd directory is owned by librenms/librenms:
This is very strange. I literally installed CentOS8 today and configured RRDcached on it, and mine is working. The only difference it seems it that I configured mine to listen on IP (for distributed setup) rather than unix socket. But other than that, they are identical.
If you run journalctl -xe do you perhaps get a bit more info? Maybe a different error message?
Content of my /etc/systemd/system/rrdcached.service
@matthewb I have the exactly same issue building the CentOS8 VM but didnt had time check what is happening so please, if you find a fix dont forget to post it here!
Thanks very much for the response. It gives me a useful data point to work on.
journalctl -xe doesn’t really add much detail unfortunately:
Jul 29 09:11:40 localhost.localdomain systemd[1]: Starting Data caching daemon for rrdtool...
-- Subject: Unit rrdcached.service has begun start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit rrdcached.service has begun starting up.
Jul 29 09:11:40 localhost.localdomain rrdcached[2572]: Failed to create base directory '/opt/librenms/rrd/': Permission denied
Jul 29 09:11:40 localhost.localdomain systemd[1]: rrdcached.service: Control process exited, code=exited status=3
Jul 29 09:11:40 localhost.localdomain systemd[1]: rrdcached.service: Failed with result 'exit-code'.
Jul 29 09:11:40 localhost.localdomain systemd[1]: Failed to start Data caching daemon for rrdtool.
-- Subject: Unit rrdcached.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit rrdcached.service has failed.
--
-- The result is failed.
On your suggestion I tried running it with it listening to an IP address rather than a unix socket but got the same error. I also tried deleting /opt/librenms/rrd and then running it but, again, same error.
After some more poking around, though, I think I might be getting closer to the issue. I think it might tripping over selinux:
I’m going to work on this a bit more and I’ll post whatever I find but a question if I may - when you did your Centos 8 install did you configure selinux as described in the LibreNMS documentation?
semanage fcontext -a -t rrdcached_exec_t '/opt/librenms/rrd/(/.*)?'
but that didn’t help. Setting rrdcached to permissive, though, did stop the error from happening and rrdcached can now start:
[root@localhost admin]# semanage permissive -a rrdcached_t
[root@localhost admin]# systemctl enable --now rrdcached.service
[root@localhost admin]# systemctl status rrdcached.service
● rrdcached.service - Data caching daemon for rrdtool
Loaded: loaded (/etc/systemd/system/rrdcached.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-07-29 15:26:40 BST; 5min ago
Process: 2199 ExecStart=/usr/bin/rrdcached -w 1800 -z 1800 -f 3600 -s librenms -U librenms -G librenms -B -R -j /var/tmp -l unix:/run/rrdcached.sock -t 4 -F -b /opt/lib>
Main PID: 2200 (rrdcached)
Tasks: 7 (limit: 11058)
Memory: 1.5M
CGroup: /system.slice/rrdcached.service
└─2200 /usr/bin/rrdcached -w 1800 -z 1800 -f 3600 -s librenms -U librenms -G librenms -B -R -j /var/tmp -l unix:/run/rrdcached.sock -t 4 -F -b /opt/librenms/rr>
Jul 29 15:26:40 localhost.localdomain systemd[1]: Starting Data caching daemon for rrdtool...
Jul 29 15:26:40 localhost.localdomain systemd[1]: Started Data caching daemon for rrdtool.
Obviously setting it to permissive is a kludge rather than a proper fix but I’m not confident enough with selinux to know the right way to tackle this properly.
Thanks. That’s the page I linked to earlier. I tried a couple of things that seemed obvious but they didn’t work. I haven’t done much with selinux so I think I’m missing something important. Setting rrdcached_t to permissive allowed it to start. I know that’s not a proper long-term fix but it’s good enough for now.
What you might want to do is to install setroubleshoot-server and then run ‘sudo sealert -a /var/log/audit/audit.log’ That helps you to find denied rules and suggest what you need to do to have it allowed.
I have a quick question about how you run rrdcached: do you only connect to it via unix socket or also via tcp socket?
The reason i’m asking is that i was also having the same problems with rrdcached and your solution of creating SElinux module works, but only when i connect via unix socket.
I get a ‘rrdcached: listen(0:42217) failed: Permission denied’ for the tcp socket.
I’m running a distributed setup, so i need to connect to my rrdcached server via tcp…
@JohnnyBBravo I run rrdcached as unix socket.
You could maybe use grep rrdcached /var/log/audit/audit.log | audit2allow -m rrdached_librenms in order to find what selinux privileges you are missing.
Please post your generated selinux .tt file for completition.
I had modified the .te file you posted above a little, but that didn’t help.
This is what i used:
module rrdcached_librenms 1.0;
require {
type httpd_t;
type httpd_sys_rw_content_t;
type rrdcached_t;
type var_run_t;
class capability { dac_read_search fsetid };
class dir { getattr search };
class file { getattr lock map open read write };
class sock_file { create setattr unlink write };
class unix_stream_socket connectto;
class capability net_raw;
class rawip_socket { getopt create setopt write read };
}
#============= httpd_t ==============
#!!!! This avc can be allowed using the boolean 'daemons_enable_cluster_mode'
allow httpd_t rrdcached_t:unix_stream_socket connectto;
allow httpd_t var_run_t:sock_file write;
allow httpd_t self:capability net_raw;
allow httpd_t self:rawip_socket { getopt create setopt write read };
#============= rrdcached_t ==============
allow rrdcached_t httpd_sys_rw_content_t:dir { getattr search };
#!!!! This avc can be allowed using the boolean 'domain_can_mmap_files'
allow rrdcached_t httpd_sys_rw_content_t:file map;
allow rrdcached_t httpd_sys_rw_content_t:file { getattr lock open read write };
allow rrdcached_t self:capability { dac_read_search fsetid };
allow rrdcached_t var_run_t:sock_file { create setattr unlink };
allow rrdcached_t self:capability net_raw;
allow rrdcached_t self:rawip_socket { getopt create setopt write read };
When i run grep rrdcached /var/log/audit/audit.log | audit2allow -m rrdached_librenms the output is:
module rrdached_librenms 1.0;
require {
type rrdcached_t;
type httpd_sys_rw_content_t;
class dir getattr;
class tcp_socket listen;
}
#============= rrdcached_t ==============
#!!!! This avc is allowed in the current policy
allow rrdcached_t httpd_sys_rw_content_t:dir getattr;
#!!!! This avc can be allowed using the boolean 'nis_enabled'
allow rrdcached_t self:tcp_socket listen;
I’m not overly familiar with SElinux, but why would the output of grep rrdcached /var/log/audit/audit.log | audit2allow -m rrdached_librenms be different to the .te file that I started with?
The answer was in the output above already…
If I do setsebool -P nis_enabled 1 then everything works fine, also with SElinux in enforcing mode!
The modifications i had made to your .te file were incorrect, but have now found the right ones.
If I use this, then all works fine, also without the setsebool -P nis_enabled.
module rrdcached_librenms 1.0;
require {
type httpd_t;
type httpd_sys_rw_content_t;
type rrdcached_t;
type var_run_t;
class capability { dac_read_search fsetid };
class dir { getattr search };
class file { getattr lock map open read write };
class sock_file { create setattr unlink write };
class unix_stream_socket connectto;
class tcp_socket { listen };
}
#============= httpd_t ==============
allow httpd_t rrdcached_t:unix_stream_socket connectto;
allow httpd_t var_run_t:sock_file write;
#============= rrdcached_t ==============
allow rrdcached_t httpd_sys_rw_content_t:dir { getattr search };
allow rrdcached_t httpd_sys_rw_content_t:file map;
allow rrdcached_t httpd_sys_rw_content_t:file { getattr lock open read write };
allow rrdcached_t self:capability { dac_read_search fsetid };
allow rrdcached_t var_run_t:sock_file { create setattr unlink };
allow rrdcached_t self:tcp_socket { listen };
Many thanks for your help, it put me on the right path to find the issue!!
JB