I’m slightly confident the ongoing reboot events are real events, unlike yesterday. Although the count is not as obviously typical as we would hope. Partly because our daily.sh runs at 00:15:00, so there is some carryover of yesterday’s false reboots in this strict date grouping.
MariaDB [librenms]> select DATE(datetime), COUNT(event_id) from eventlog where type='reboot' and datetime>="2021-10-01 00:00:00" group by DATE(datetime) order by DATE(datetime);
+----------------+-----------------+
| DATE(datetime) | COUNT(event_id) |
+----------------+-----------------+
| 2021-10-01 | 445 |
| 2021-10-02 | 397 |
| 2021-10-03 | 388 |
| 2021-10-04 | 433 |
| 2021-10-05 | 420 |
| 2021-10-06 | 372 |
| 2021-10-07 | 430 |
| 2021-10-08 | 470 |
| 2021-10-09 | 376 |
| 2021-10-10 | 423 |
| 2021-10-11 | 435 |
| 2021-10-12 | 3160 |
| 2021-10-13 | 692 |
| 2021-10-14 | 394 |
| 2021-10-15 | 392 |
| 2021-10-16 | 362 |
| 2021-10-17 | 195 |
| 2021-10-18 | 107 |
| 2021-10-19 | 467 |
| 2021-10-20 | 84807 |
| 2021-10-21 | 1055 |
+----------------+-----------------+
21 rows in set (1.535 sec)
The vast majority of our devices are Motorola Cambium radios. The friendly/human uptime listed in the device webui (http…/device/device_id) matches on the device itself. However, the message field in eventlog table doesn’t agree, and that message is presented in both Recent Events on the device webui and the webui eventlog. Overlooking device_id 210 and 211 (which have always shown false reboots, thankfully we have very few Mimosa radios), this looks pretty clear that the friendly/human portion of the message is getting calculated from the reference, which contradicts the “-> XXs” in the message itself.
MariaDB [librenms]> select * from eventlog where type='reboot' and datetime>="2021-10-21 00:00:00" order by event_id DESC limit 15;
+----------+-----------+---------------------+----------------------------------------------------------------------+--------+-----------+----------+----------+
| event_id | device_id | datetime | message | type | reference | username | severity |
+----------+-----------+---------------------+----------------------------------------------------------------------+--------+-----------+----------+----------+
| 9075452 | 2619 | 2021-10-21 10:32:40 | Device rebooted after 2 hours 33 minutes 35 seconds -> 360s | reboot | 9215 | | 4 |
| 9075430 | 211 | 2021-10-21 10:30:03 | Device rebooted after 1 hour 49 minutes 37 seconds -> 275s | reboot | 6577 | | 4 |
| 9075423 | 2667 | 2021-10-21 10:26:35 | Device rebooted after 34 minutes 49 seconds -> 219s | reboot | 2089 | | 4 |
| 9075329 | 210 | 2021-10-21 10:10:03 | Device rebooted after 3 minutes 53 seconds -> 223s | reboot | 233 | | 4 |
| 9075328 | 2260 | 2021-10-21 10:09:01 | Device rebooted after 5 days 17 hours 48 minutes 33 seconds -> 290s | reboot | 496113 | | 4 |
| 9075295 | 210 | 2021-10-21 10:05:02 | Device rebooted after 4 minutes 5 seconds -> 233s | reboot | 245 | | 4 |
| 9075291 | 2326 | 2021-10-21 10:03:52 | Device rebooted after 13 hours 2 minutes 23 seconds -> 110s | reboot | 46943 | | 4 |
| 9075269 | 2076 | 2021-10-21 10:00:24 | Device rebooted after 2 days 17 hours 1 minute 25 seconds -> 296s | reboot | 234085 | | 4 |
| 9075259 | 210 | 2021-10-21 10:00:02 | Device rebooted after 14 minutes 34 seconds -> 245s | reboot | 874 | | 4 |
| 9075249 | 1596 | 2021-10-21 09:56:46 | Device rebooted after 9 days 1 hour 42 minutes 13 seconds -> 42s | reboot | 783733 | | 4 |
| 9075244 | 2576 | 2021-10-21 09:55:50 | Device rebooted after 3 hours 30 minutes 2 seconds -> 140s | reboot | 12602 | | 4 |
| 9075223 | 2544 | 2021-10-21 09:53:39 | Device rebooted after 6 hours 55 minutes 39 seconds -> 90s | reboot | 24939 | | 4 |
| 9075173 | 2292 | 2021-10-21 09:47:42 | Device rebooted after 1 day 48 minutes 15 seconds -> 309s | reboot | 89295 | | 4 |
| 9075150 | 210 | 2021-10-21 09:45:02 | Device rebooted after 8 minutes 46 seconds -> 272s | reboot | 526 | | 4 |
| 9075146 | 2667 | 2021-10-21 09:44:00 | Device rebooted after 3 hours 19 minutes 48 seconds -> 72s | reboot | 11988 | | 4 |
+----------+-----------+---------------------+----------------------------------------------------------------------+--------+-----------+----------+----------+
15 rows in set (0.014 sec)