It’s the same as snmp. ICMP checks are only done at poll time.
Use services if you want something more.