LibreNMS rrdcached running but cant connect by poller

Hello All,

I have this setting on my libreNMS:
Pacemaker cluster:

  • 2 DB server consist of Mariadb and rrdcached using drbd (Distributed Replicated Block Device), have memcached also
  • 2 Libre web and 38 poller machine connected to DB server on same network

Everything seems OK when poller started to connect.
Sometimes, the librenms poller cant connect to the rrdcached, it has error like this:

rrdcached[42353]: listen_thread_main: accept(2) failed.

The total TCP connection at that time also high, its around 6000 connections.

In order to fix it, I need to turn off all poller and restart the rrdcached service which is not good for production environment.

Anyone experience it before? Please help. Thanks.