LibreNMS polling stops when REDIS is enabled in .env

Tags: #<Tag:0x00007fb5ce8dae60>

Hi,

I get this error page when I enable redis in the .env for librenms. Following is the redis script that I have included. Polling stopped as soon as redis is included.

REDIS_HOST=10.28.132.61
REDIS_PORT=6379
REDIS_DB=0
CACHE_DRIVER=redis

(Attached is the screenshot of the web-gui error)

Following is the validate output from librenms.

[[email protected] ksantosh]# su - librenms
Last login: Tue Jul 20 12:09:16 IST 2021 on pts/0
-bash-4.2$ ./validate.php

Component Version
LibreNMS 21.7.0-8-ga45020c
DB Schema 2021_25_01_0127_create_isis_adjacencies_table (211)
PHP 7.3.29
Python 3.6.8
MySQL 10.4.20-MariaDB
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 2.1.3
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct
[INFO] Detected Python Wrapper
[FAIL] We could not get memcached stats, it is possible that we cannot connect to your memcached server, please check
[WARN] Your local git contains modified files, this could prevent automatic updates.
[FIX]:
You can fix this with ./scripts/github-remove
Modified Files:
bootstrap/cache/.gitignore
logs/.gitignore
rrd/.gitignore
storage/app/.gitignore
storage/app/public/.gitignore
storage/debugbar/.gitignore
storage/framework/cache/.gitignore
storage/framework/cache/data/.gitignore
storage/framework/sessions/.gitignore
storage/framework/testing/.gitignore
storage/framework/views/.gitignore
storage/logs/.gitignore

Did you check the logs like it says?

Yes Sir. I have taken a snippet of the logs like it said and it’s a lengthy notepad. I just don’t know how to upload it here.

I didn’t find anything significant there other than the messages that it has polled devices in so many seconds and one entry where it says that it has polled zero devices.

Thanks.

Please find a snippet of the logs here:

https://p.libren.ms/view/624db7c8

Thanks,
Santosh Kotla

[2021-07-20 23:09:28] production.ERROR: Permission denied [tcp://10.28.132.61:6379]

I’m guessing you have selinux enabled

I just went through the config procedure as for Centos and I see that there is section where SELinux is configured. I am using CentOS 7 but the one mentioned in the installation manual is for CentOS 8. Do you think this would pose a problem?

Could SELinux be an issue?

Thanks,

what does your redis.conf and .env look like? I didn’t set redis_db=0 in my .env file. I am running redis/rrdcached/galera on centos 7 with 4 pollers.

I haven’t changed anything in the redis.conf. Just commented the bind statement so that the tool uses all network ports.

Below is the snippet of the .env file.

[[email protected] ksantosh]# cat /opt/librenms/.env
APP_KEY=base64:bUA62mcsJbBD1cTVaXISeiJjqGyUx8kRI8VYpjNv5ek=

DB_HOST=localhost
DB_DATABASE=librenms
DB_USERNAME=librenms
DB_PASSWORD=librenms

#REDIS_HOST=10.28.132.61
#REDIS_PORT=6379
#REDIS_DB=0
#CACHE_DRIVER=redis

#APP_URL=
NODE_ID=60eebcb42118b
LIBRENMS_USER=librenms

Thanks.

P.S - I overlooked that CentOS 8 was mentioned in the install document and pushed for a CentOS 8 upgrade last night.

[[email protected] ksantosh]# cat /etc/centos-release
CentOS Linux release 8.4.2105

So 10.28.132.61 is the master or is it standalone. You can try this. X.X.X.X/32 will be the IP address of the redis client/replica/slave.

If you think it is selinux or firewalld, you can disable them and restart your httpd/nginx and redis service.

systemctl stop firewalld
setenforce 0

Or you can keep the firewall enabled and add a rule for it and also add policies for selinux.

systemctl start firewalld
firewall-cmd --permanent --zone=public --add-port=6379/tcp
firewall-cmd --permanent --zone=public --add-source=X.X.X.X/32

sudo firewall-cmd --reload
systemctl stop firewalld

semanage port -a -t redis_port_t -p tcp 6379

semanage permissive -a redis_t

setsebool -P httpd_can_network_connect=1

systemctl restart httpd
systemctl start firewalld

php -m | grep redis

Thanks for the detailed suggestion nms.

Do you suggest that I change the REDIS_HOST to 10.28.132.61/32?

I don’t think the SELinux is the culprit because this issue was happening even when SELinux was disabled. Firewall has been disabled as well, as this instance will be working only in a private environment.

Scenario::
I am testing this before I deploy a poller in our PROD environment. We have a Librenms instance that is taking care of about 1300 devices in a DC in Reno. At present we are maxing out on the resources for this instance and the polling of the devices are not completing. I am planning to deploy poller in APAC and EMEA so that the respective devices in those geographies can be offloaded to the poller of that region.

This test instance I deployed is supposed to be a mock central server and then I will deploy a poller that offloads specified devices from this mock central server.

I hope you got an idea of what I am trying to achieve.

I don’t how 10.28.132.61/32 relates to your environment. Do you have only a single librenms instance running right now?

Yes. This is the only instance that I have right now and I am trying to st this up as a mock central server. Once this comes up fine, I will configure pollers to offload some work from the mock central server.

Thanks,
Santosh Kotla