[Solve] Long Polling Duration since about one week

Help
1

It’s very strange, since about more than one week, the polling duration of all my centos linux server is near the 300s

poller.png1135×252 10.9 KB

But when I launch the poller from command line it’s only take 5s :

-bash-4.2$ time ./poller.php -h dmcvypft001
LibreNMS Poller
Version info:
Commit SHA: 3cd3a3262a28eb918eb24594347845f7c0b7e5c3
Commit Date: 1485904509
DB Schema: 157
PHP: 7.0.15
MySQL: 10.1.21-MariaDB
RRDTool: 1.4.8
SNMP: NET-SNMP 5.7.2
Starting polling run:

Hostname: dmcvypft001
Device ID: 83
OS: linux (unix)

Using hrSystemUptime (3123773s)
Uptime: 36 days, 3h 42m 53s
Module [ unix-agent ] disabled on host.

Load poller module os

Hardware: Generic x86 64-bit
Version: 2.6.32-642.11.1.el6.x86_64
Features:
Serial:

Runtime for poller module ‘os’: 0.9369 seconds

Unload poller module os

Module [ ipmi ] disabled globally.

Module [ sensors ] disabled globally.

Load poller module processors

Processor AMD Opteron™ Processor 6164 HE… 1%
Processor AMD Opteron™ Processor 6164 HE… 1%
Processor AMD Opteron™ Processor 6164 HE… 1%
Processor AMD Opteron™ Processor 6164 HE… 0%

Runtime for poller module ‘processors’: 0.1158 seconds

Unload poller module processors

Load poller module mempools

Mempool Physical memory: 37.03%
Mempool Virtual memory: 29.22%
Mempool Swap space: 0.01%

Runtime for poller module ‘mempools’: 0.0473 seconds

Unload poller module mempools

Load poller module storage

Storage /: 16%
Storage /dev/shm: 0%
Storage /boot: 42%
Storage /tmp: 0%
Storage /usr: 45%
Storage /var: 34%

Runtime for poller module ‘storage’: 0.0520 seconds

Unload poller module storage

Load poller module netstats

IP TCP UDP ICMP SNMP IP-FORWARD

Runtime for poller module ‘netstats’: 0.2300 seconds

Unload poller module netstats

Load poller module hr-mib

Processes Users

Runtime for poller module ‘hr-mib’: 0.0287 seconds

Unload poller module hr-mib

Load poller module ucd-mib

Runtime for poller module ‘ucd-mib’: 0.0796 seconds

Unload poller module ucd-mib

Load poller module ipSystemStats

ipv4 ipv6

Runtime for poller module ‘ipSystemStats’: 0.0379 seconds

Unload poller module ipSystemStats

Load poller module ports

Caching Oids: ifDescr ifAdminStatus ifOperStatus ifLastChange ifType ifPhysAddress ifMtu ifInErrors ifOutErrors ifInDiscards ifOutDiscards dot3StatsDuplexStatusvalidvalidvalid
Port lo: lo (1 / #274) HC HC VLAN == lobps(407.000bps/407.000bps)bytes(5.310kB/5.310kB)pkts(0.520pps/0.520pps)
Port eth0: eth0 (2 / #275) HC HC dot3Duplex VLAN == eth0bps(545.000bps/6.640kbps)bytes(7.120kB/86.730kB)pkts(0.710pps/1.010pps)
Port eth1: eth1 (3 / #276) HC HC dot3Duplex VLAN == eth1bps(1.900kbps/560.000bps)bytes(24.810kB/7.320kB)pkts(0.710pps/0.640pps)

Runtime for poller module ‘ports’: 1.5973 seconds

Unload poller module ports

Module [ bgp-peers ] disabled on os.

Module [ junose-atm-vp ] disabled globally.

Module [ toner ] disabled globally.

Load poller module ucd-diskio

sda sda1 sda2 dm-0 dm-1 dm-2 dm-3 sda

Runtime for poller module ‘ucd-diskio’: 0.0817 seconds

Unload poller module ucd-diskio

Module [ wifi ] disabled globally.

Module [ ospf ] disabled on os.

Module [ cisco-ipsec-flow-monitor ] disabled globally.

Module [ cisco-remote-access-monitor ] disabled globally.

Module [ cisco-cef ] disabled globally.

Module [ cisco-sla ] disabled globally.

Module [ cisco-mac-accounting ] disabled globally.

Module [ cipsec-tunnels ] disabled globally.

Module [ cisco-ace-loadbalancer ] disabled globally.

Module [ cisco-ace-serverfarms ] disabled globally.

Module [ cisco-asa-firewall ] disabled globally.

Module [ cisco-voice ] disabled globally.

Module [ cisco-cbqos ] disabled globally.

Module [ cisco-otv ] disabled globally.

Module [ netscaler-vsvr ] disabled globally.

Module [ aruba-controller ] disabled globally.

Module [ entity-physical ] disabled globally.

Load poller module applications

apache mailscanner

Runtime for poller module ‘applications’: 0.2164 seconds

Unload poller module applications

Load poller module mib

MIB: polling acldropCounter dropCounter KNIdropCounter KNIrxCounter KNItxCounter ratedropCounter rxCounter txCounter

Runtime for poller module ‘mib’: 0.2985 seconds

Unload poller module mib

Module [ stp ] disabled on os.

Load poller module ntp

Runtime for poller module ‘ntp’: 0.0001 seconds

Unload poller module ntp

Load poller module services

Runtime for poller module ‘services’: 0.0001 seconds

Unload poller module services

Module [ loadbalancers ] disabled globally.

uptime netstat_ip netstat_ip_frag netstat_tcp netstat_udp netstat_icmp netstat_icmp_info netstat_snmp netstat_snmp_pkt hr_processes hr_users ucd_cpu ucd_swap_io ucd_io ucd_contexts ucd_interrupts ucd_memory ucd_load ipsystemstats_ipv4 ipsystemstats_ipv4_frag ipsystemstats_ipv6 ipsystemstats_ipv6_frag Polled in 4.530 seconds
UPDATED!

Start Alerts

Rule #2 (Device rebooted): Status: NOCHG
Rule #9 (Service up/down): Status: NOCHG
Rule #10 (swap usage): Status: NOCHG

End Alerts

SNMP: Get[22/0.82s] Walk [30/2.94s]
MySQL: Cell[4/0.00s] Row[6/0.00s] Rows[33/0.01s] Column[3/0.00s] Update[20/0.03s] Insert[2/0.00s] Delete[0/0.00s]

real 0m4.718s
user 0m0.933s
sys 0m0.612s

2

What if you add the debug statement?
./poller.php -h dmcvypft001 -d

8

Please for the love of god pastebin the output!

9

sorry,

the pastbin link

Vincent

10

so If I launch the poller from commande line every thing is OK
but I the poller is launch by the poller-warpper, I have the long polling PB.
I do an strace on the process launch by pooler-wrapper :
librenms 12247 12245 0 15:10 ? 00:00:00 php /data/librenms/poller.php -h 83
librenms 13735 2401 0 15:11 pts/0 00:00:00 grep --color=auto poll
-bash-4.2$ strace -p12247
Process 12247 attached
restart_syscall(<… resuming interrupted call …>) = 0
poll([{fd=4, events=POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)
poll([{fd=4, events=POLLOUT}], 1, 1000) = 0 (Timeout)
poll([{fd=4, events=POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)
poll([{fd=4, events=POLLOUT}], 1, 1000) = 0 (Timeout)
poll([{fd=4, events=POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)
poll([{fd=4, events=POLLOUT}], 1, 1000) = 0 (Timeout)
poll([{fd=4, events=POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)
poll([{fd=4, events=POLLOUT}], 1, 1000) = 0 (Timeout)
poll([{fd=4, events=POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)
poll([{fd=4, events=POLLOUT}], 1, 1000) = 0 (Timeout)
poll([{fd=4, events=POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)
poll([{fd=4, events=POLLOUT}], 1, 1000) = 0 (Timeout)
poll([{fd=4, events=POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)
poll([{fd=4, events=POLLOUT}], 1, 1000) = 0 (Timeout)
poll([{fd=4, events=POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)
poll([{fd=4, events=POLLOUT}], 1, 1000) = 1 ([{fd=4, revents=POLLOUT|POLLERR|POLLHUP}])
poll([{fd=4, events=POLLOUT|POLLWRNORM}], 1, 0) = 1 ([{fd=4, revents=POLLOUT|POLLWRNORM|POLLERR|POLLHUP}])
getsockopt(4, SOL_SOCKET, SO_ERROR, [110], [4]) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 10
fcntl(10, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(10, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(10, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr(“216.58.204.234”)}, 16) = -1 EINPROGRESS (Operation now in progress)
close(4) = 0
poll([{fd=10, events=POLLOUT}], 1, 1000) = 0 (Timeout)
poll([{fd=10, events=POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)

11

OK, I found the mistake/bug

The librenms server have no internet connexion.
The servers who have the long time pooling have a syslocation define.

I find that each time the poller is launch by the wrapper, he try to connect to googlemap I think and then update the location of the server but If we have no internet connexion he fall in timeout.

I enable the proxy setting in librenms to use a proxy and the long time polling is solved.

Vincent

PS: do we have a way to diable this googlemaps connection ?

12

Set $config['geoloc']['latlng']=false;

13

Thanks, it work

Vincent

14

I’ll push another fix later that would timeout the curl requests so this should be less of an issue.