Poller Setup Issues

Hi Team,

I am trying to offload some of the compute from the Central Server by deploying a Poller in the Environment.

I have deployed them on Ubuntu 20.04 LTS and I didn’t have any issues in the install. REDIS, RRDTool and MEMCACHED have been deployed on the central server but there seems to be issues when the poller setup happens. They don’t seem to communicate right and I need some help in sorting the issues out.

The Validates have been done and they come out with no major failures. Attached are the screenshots for the same. Any suggestions from anyone is highly appreciated.

Thanks,
Santosh Kotla

Central Server

Poller
image

Some more stats from the background:

[email protected]:~# systemctl status rrdcached
● rrdcached.service - LSB: start or stop rrdcached
Loaded: loaded (/etc/init.d/rrdcached; generated)
Active: active (running) since Tue 2021-10-19 10:35:49 UTC; 2h 38min ago
Docs: man:systemd-sysv-generator(8)
Tasks: 7 (limit: 19108)
Memory: 5.9M
CGroup: /system.slice/rrdcached.service
└─842788 /usr/bin/rrdcached -l 0:42217 -R -j /var/lib/rrdcached/journal/ -F -b /opt/librenms/rrd -B -w 1800 -z 900 -w 1800 -z 1800 -t 4 -b /opt/librenms/rrd/ -j /var/lib/rrdcached/journal/ -G librenms -U librenms -p /run/r>

Oct 19 10:35:49 Test-ksantosh01-Ubuntu20 systemd[1]: Stopped LSB: start or stop rrdcached.
Oct 19 10:35:49 Test-ksantosh01-Ubuntu20 systemd[1]: Starting LSB: start or stop rrdcached…
Oct 19 10:35:49 Test-ksantosh01-Ubuntu20 rrdcached[842787]: rrdcached: can’t create pid file ‘/run/rrdcached.pid’ (File exists)
Oct 19 10:35:49 Test-ksantosh01-Ubuntu20 rrdcached[842787]: rrdcached: removed stale PID file (no rrdcached on pid 720040)
Oct 19 10:35:49 Test-ksantosh01-Ubuntu20 rrdcached[842787]: rrdcached: starting normally.
Oct 19 10:35:49 Test-ksantosh01-Ubuntu20 rrdcached[842766]: * rrdcached started
Oct 19 10:35:49 Test-ksantosh01-Ubuntu20 systemd[1]: Started LSB: start or stop rrdcached.

[email protected]:~# netstat -ant4 | grep 42217
tcp 0 0 0.0.0.0:42217 0.0.0.0:* LISTEN

[email protected]:~# systemctl status librenms
● librenms.service - LibreNMS SNMP Poller Service
Loaded: loaded (/etc/systemd/system/librenms.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-10-19 13:15:15 UTC; 754ms ago
Main PID: 867138 ((rvice.py))
Tasks: 0 (limit: 19108)
Memory: 0B
CGroup: /system.slice/librenms.service
└─867138 (rvice.py)

Oct 19 13:15:15 Test-ksantosh01-Ubuntu20 systemd[1]: Started LibreNMS SNMP Poller Service.

[email protected]:~# systemctl status redis
● redis-server.service - Advanced key-value store
Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-10-19 09:53:00 UTC; 3h 22min ago
Docs: http://redis.io/documentation,
man:redis-server(1)
Main PID: 836591 (redis-server)
Tasks: 4 (limit: 19108)
Memory: 14.2M
CGroup: /system.slice/redis-server.service
└─836591 /usr/bin/redis-server 10.28.132.61:6379

Oct 19 09:52:37 Test-ksantosh01-Ubuntu20 systemd[1]: Starting Advanced key-value store…
Oct 19 09:53:00 Test-ksantosh01-Ubuntu20 systemd[1]: redis-server.service: Can’t open PID file /run/redis/redis-server.pid (yet?) after start: Operation not permitted
Oct 19 09:53:00 Test-ksantosh01-Ubuntu20 systemd[1]: Started Advanced key-value store.

Thanks,
Santosh Kotla

Hi @murrant @Jellyfrog @paulierco @kalamchi75

Can you please help me understand what could be stopping the services on my central instance. I have deployed both the poller and the central server on Ubuntu 20.04 LTS.

Please let me know if you need some stats from my side.

Thanks,
Santosh Kotla

Hi Santosh,

redis-server.service: Can’t open PID file /run/redis/redis-server.pid (yet?) after start: Operation not permitted

who owns the file ? looks like a permission issue. In the worst case, change the file mod to 777 (permission to everything for all) and see
however, before 777, investigate which user owns that file, and which should own it for redis to work properly.

Hi @kalamchi75

I have checked on the permission as suggested earlier but they seem to be in place. The redis folder and it’s files are always owned by REDIS.

[email protected]:~$ ls -l /run/redis/
total 4
-rw-rw---- 1 redis redis 6 Oct 25 19:16 redis-server.pid

How do we apply to change mod to 777 on the redis folder?

Thanks,
Santosh Kotla

hi Santosh,

just apply it for the file being complained about:

chmod 777 /run/redis/redis-server.pid

Hi @kalamchi75

Thanks for the command. I have applied it and hit the redis service for a restart:

[email protected]:~# systemctl status redis
● redis-server.service - Advanced key-value store
Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-10-27 12:58:16 UTC; 10s ago
Docs: http://redis.io/documentation,
man:redis-server(1)
Process: 1229373 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf (code=exited, status=0/SUCCESS)
Main PID: 1229398 (redis-server)
Tasks: 5 (limit: 19107)
Memory: 9.0M
CGroup: /system.slice/redis-server.service
└─1229398 /usr/bin/redis-server *:6379

Oct 27 12:58:16 Test-ksantosh01-Ubuntu20 systemd[1]: Starting Advanced key-value store…
Oct 27 12:58:16 Test-ksantosh01-Ubuntu20 systemd[1]: redis-server.service: Can’t open PID file /run/redis/redis-server.pid (yet?) after start: Operation not permitted
Oct 27 12:58:16 Test-ksantosh01-Ubuntu20 systemd[1]: Started Advanced key-value store.

It still throws me the same error. Just for your validation, below is a snippet of the redis version I am using.

[email protected]:~# redis-server -v
Redis server v=6.0.6 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=d463d609620685c0

Thanks,
Santosh Kotla

Hi @kalamchi75

Sir. Can you please tell me if you have deployed the dispatcher service in your environment or if you are using with the cron?

Thanks,
Santosh Kotla

Hi @santoshkotla

How are you doing my friend ? I hope all fine.

No, not really. I always use the cron.

Best Regards

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.