New Traffic Bill polling fails / long bill-calculate.php run time

Help
1

I’m running into an issue where every Traffic Bill with bill_id > 400 is not storing historical data or calculating utilization percentages. It’s showing traffic graphs for Billing, 24 Hour, and Monthly views – but that is all. Has anyone else run into this?

Edit: On LibreNMS 26.3.0 but I believe this has been occurring for months. The only error I am seeing in log is this.

Edit 2: This appears to be due to bill-calculate.php running longer than 3600 seconds and timing out.

[2026-03-18T08:09:39][CRITICAL] Exception: DivisionByZeroError Division by zero @ /opt/librenms/incl
udes/functions.php:349
#0 /opt/librenms/poll-billing.php(80): delta_to_bits()
#1 {main}


librenms@libre:~$ ./validate.php
===========================================
Component | Version
--------- | -------
LibreNMS  | 26.3.0 (2026-03-16T04:02:39-07:00)
DB Schema | 2026_03_05_112733_ospfv3_instances_nullable (374)
PHP       | 8.3.17
Python    | 3.12.3
Database  | MariaDB 10.11.14-MariaDB-0ubuntu0.24.04.1
RRDTool   | 1.7.2
SNMP      | 5.9.4.pre2
===========================================

[OK]    Composer Version: 2.9.5
[OK]    Dependencies up-to-date.
[OK]    Database Connected
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    Database and column collations are correct
[OK]    Database schema correct
[OK]    MySQL and PHP time match
[OK]    Active pollers found
[OK]    Dispatcher Service is enabled
[OK]    Locks are functional
[OK]    Python wrapper cron entry is not present
[OK]    Redis is unavailable
[OK]    rrdtool version ok
[OK]    Connected to rrdcached
2

The over 400 must be a red herring as I’ve got a client with bills over that with no issues.

Distributed polling at all?

3

Single poller. Thanks for letting me know. I’ll dig further.

4

Seems that delta_to_bits in functions.php is being fed $period == 0 from time to time by poll-billing.php and it borks. I put in a nasty hack in delta_to_bits and poll-billing.php is completing. I’ll see if data starts populating now.

function delta_to_bits($delta, $period)
{
    if ($period == 0)
        return 0;
    return round($delta * 8 / $period, 2);
}
5

My hack appears to be working. Why delta_to_bits is being fed a zero is another question.

MariaDB [librenms]> select * from bill_data where bill_id = 415;
+----------+---------+---------------------+--------+---------------+---------------+---------------+
| id       | bill_id | timestamp           | period | delta         | in_delta      | out_delta     |
+----------+---------+---------------------+--------+---------------+---------------+---------------+
| 31308540 |     415 | 2026-03-18 11:05:00 |      0 |             0 |             0 |             0 |
| 31308878 |     415 | 2026-03-18 11:29:02 |   1442 | 2358382517270 | 1147959170076 | 1210423347194 |
+----------+---------+---------------------+--------+---------------+---------------+---------------+
2 rows in set (0.000 sec)
6

It appears that, when creating a new bill, period is set to zero. I’d think this would be causing issues for any new bills. SQL query immediately after creating a bill, then after manually running poll-billing.php (with my hack in place).

MariaDB [librenms]> select * from bill_data where bill_id = 416;
Empty set (0.001 sec)

MariaDB [librenms]> select * from bill_data where bill_id = 416;
+----------+---------+---------------------+--------+-------+----------+-----------+
| id       | bill_id | timestamp           | period | delta | in_delta | out_delta |
+----------+---------+---------------------+--------+-------+----------+-----------+
| 31309555 |     416 | 2026-03-18 12:57:47 |      0 |     0 |        0 |         0 |
+----------+---------+---------------------+--------+-------+----------+-----------+
1 row in set (0.001 sec)
8

Another red herring. That function hasn’t changed for 10 years, nor has the invoking of it.

I’ve just added a bill and 0 for period is correct for the first run of poll-billing, then it will populate:

MariaDB [librenms]> select * from bill_data where bill_id = 6;

±------±--------±--------------------±-------±--------±---------±----------+

| id    | bill_id | timestamp           | period | delta   | in_delta | out_delta |

±------±--------±--------------------±-------±--------±---------±----------+

| 67660 |       6 | 2026-03-19 00:05:00 |      0 |       0 |        0 |         0 |

| 67663 |       6 | 2026-03-19 00:10:00 |    300 | 1643872 |   853620 |    790252 |

±------±--------±--------------------±-------±--------±---------±----------+
9

Hmm. delta_to_bits is being handed that $period = 0 and PHP is throwing an exception. I’d expect that for a divide by zero.

From delta_to_bits –

round($delta * 8 / $period, 2)

From poll-billing.php it looks like $period is calculated thus –

$tmp_period = dbFetchCell(“SELECT UNIX_TIMESTAMP(CURRENT_TIMESTAMP()) - UNIX_TIMESTAMP('” . $last_counters[‘timestamp’] . “')”);

I’ll poke at it some more.

10

It is bill-calculate.php that initializes bill_data in a new bill? I’m seeing bill-calculate.php running for a whole hour, then timing out. I’m wondering if it makes it to around the 400th bill or so.

Also, does bill_data ever get purged? It looks like I have over five years of data for quite a few bills. I wonder if the size of that table is causing the long duration run of bill-calculate.php.

e.g.,

MariaDB [librenms]> select * from bill_data where bill_id = 5 limit 5;
+--------+---------+---------------------+--------+-------------+-------------+-------------+
| id     | bill_id | timestamp           | period | delta       | in_delta    | out_delta   |
+--------+---------+---------------------+--------+-------------+-------------+-------------+
| 669401 |       5 | 2020-12-25 10:45:03 |      0 |           0 |           0 |           0 |
| 669402 |       5 | 2020-12-25 10:50:04 |    301 | 56107722590 | 21694242589 | 34413480001 |
| 669403 |       5 | 2020-12-25 10:55:05 |    301 | 52307830721 | 21115787091 | 31192043630 |
| 669404 |       5 | 2020-12-25 11:00:03 |    298 | 55581513655 | 21589449051 | 33992064604 |
| 669405 |       5 | 2020-12-25 11:05:02 |    299 | 56077372312 | 22897138099 | 33180234213 |
+--------+---------+---------------------+--------+-------------+-------------+-------------+
5 rows in set (0.022 sec)

MariaDB [librenms]> select count(*) from bill_data where bill_id = 5;
+----------+
| count(*) |
+----------+
|   517140 |
+----------+
1 row in set (0.342 sec)

bill_data has over 23 million records

MariaDB [librenms]> select count(*) from bill_data;
+----------+
| count(*) |
+----------+
| 23413287 |
+----------+
1 row in set (14.072 sec)
11

@laf should bill-calculate be pulling all of the last 24 periods? That appears to be the case for me – and after the fourth or so period on a given bill the response time skyrockets. e.g., Using debug flag shows it pulling Feb, Jan, Dec bill_history after the ‘Updated history!’

SQL[SELECT * FROM `bills` ORDER BY `bill_id` [] 4.67ms]

3 BACKBONE:XXXXXXXXXX
SQL[SELECT * FROM `bill_history` WHERE bill_id = ? AND bill_datefrom = ? AND bill_dateto = ? LIMIT 1 [3,"20260301000000","20260331235959"] 1.05ms]

SQL[SELECT SUM(period) as `period`, MAX(in_delta) as `peak_in`, MAX(out_delta) as `peak_out`  FROM bill_data WHERE bill_id = ? AND timestamp > ? AND timestamp <= ? [3,"20260301000000","20260331235959"] 0.14ms]

SQL[SELECT SUM(period) as period, SUM(delta) as total, SUM(in_delta) as inbound, SUM(out_delta) as outbound FROM bill_data WHERE bill_id = ? AND timestamp > ? AND timestamp <= ? [3,"20260301000000","20260331235959"] 0.1ms]

SQL[SELECT (SUM(in_delta) / SUM(period) * 8) as rate, FROM_UNIXTIME(FLOOR(UNIX_TIMESTAMP(`timestamp`) / 300) * 300) AS bucket_start,   DATE_ADD(FROM_UNIXTIME(FLOOR(UNIX_TIMESTAMP(`timestamp`) / 300) * 300), INTERVAL 5 MINUTE) AS bucket_end,SUM(in_delta) as delta_sum FROM bill_data WHERE bill_id = ? AND timestamp > ? AND timesta
mp <= ? GROUP BY bill_id, bucket_start ORDER BY rate ASC [3,"20260301000000","20260331235959"] 0.34ms]

SQL[SELECT (SUM(out_delta) / SUM(period) * 8) as rate, FROM_UNIXTIME(FLOOR(UNIX_TIMESTAMP(`timestamp`) / 300) * 300) AS bucket_start,   DATE_ADD(FROM_UNIXTIME(FLOOR(UNIX_TIMESTAMP(`timestamp`) / 300) * 300), INTERVAL 5 MINUTE) AS bucket_end,SUM(out_delta) as delta_sum FROM bill_data WHERE bill_id = ? AND timestamp > ? AND times
tamp <= ? GROUP BY bill_id, bucket_start ORDER BY rate ASC [3,"20260301000000","20260331235959"] 0.31ms]

2026-03-01 00:00:00 to 2026-03-31 23:59:59 CDR      20 Gbps    428.04 Mbps 2.14%SQL[UPDATE `bills` set `rate_95th`=?,`rate_95th_in`=?,`rate_95th_out`=?,`dir_95th`=?,`total_data`=?,`total_data_in`=?,`total_data_out`=?,`rate_average`=?,`rate_average_in`=?,`rate_average_out`=?,`bill_last_calc`=NOW() WHERE `bill_id` = ? [4280
37634.71,269459233.39,428037634.71,"out","68821374566643","30853948962571","37967425604072",326635352.00173706,146436925.2066306,180198426.7951065,3] 0.19ms]

 Updated! SQL[UPDATE `bill_history` set `rate_95th`=?,`rate_95th_in`=?,`rate_95th_out`=?,`dir_95th`=?,`rate_average`=?,`rate_average_in`=?,`rate_average_out`=?,`traf_total`=?,`traf_in`=?,`traf_out`=?,`bill_peak_out`=?,`bill_peak_in`=?,`bill_used`=?,`bill_overuse`=?,`bill_percent`=?,`updated`=NOW() WHERE `bill_hist_id` = ?
[428037634.71,269459233.39,428037634.71,"out",326635352.00173706,146436925.2066306,180198426.7951065,"68821374566643","30853948962571","37967425604072",511902204310,399480935322,428037634.71,"0",2.14,5733] 0.23ms]

 Updated history!

SQL[SELECT * FROM `bill_history` WHERE bill_id = ? AND bill_datefrom = ? AND bill_dateto = ? LIMIT 1 [3,"20260201000000","20260228235959"] 0.21ms]

SQL[SELECT SUM(period) as `period`, MAX(in_delta) as `peak_in`, MAX(out_delta) as `peak_out`  FROM bill_data WHERE bill_id = ? AND timestamp > ? AND timestamp <= ? [3,"20260201000000","20260228235959"] 0.1ms]

SQL[SELECT * FROM `bill_history` WHERE bill_id = ? AND bill_datefrom = ? AND bill_dateto = ? LIMIT 1 [3,"20260101000000","20260131235959"] 0.95ms]

SQL[SELECT SUM(period) as `period`, MAX(in_delta) as `peak_in`, MAX(out_delta) as `peak_out`  FROM bill_data WHERE bill_id = ? AND timestamp > ? AND timestamp <= ? [3,"20260101000000","20260131235959"] 0.11ms]

SQL[SELECT * FROM `bill_history` WHERE bill_id = ? AND bill_datefrom = ? AND bill_dateto = ? LIMIT 1 [3,"20251201000000","20251231235959"] 0.88ms]

SQL[SELECT SUM(period) as `period`, MAX(in_delta) as `peak_in`, MAX(out_delta) as `peak_out`  FROM bill_data WHERE bill_id = ? AND timestamp > ? AND timestamp <= ? [3,"20251201000000","20251231235959"] 0.12ms]

That appears to be the root cause for everything.

12

Yes, bill-calculate re-calculates the bills based on the bill_data (iirc).

13

Thanks. Traced it down to ‘$period = Billing::getPeriod($bill[‘bill_id’], $datefrom, $dateto);’, which gets very expensive for my database if going back more than four months for some reason. e.g,

SQL[SELECT SUM(period) as `period`, MAX(in_delta) as `peak_in`, MAX(out_delta) as `peak_out`  FROM bill_data WHERE bill_id = ? AND timestamp > ? AND timestamp <= ? [19,"20251001000000","20251031235959"] 2542.81ms]

Interestingly when bill-calculate.php iterates to the next bill, response time drops again to sub 10 ms range for the first four months of billing history.

For the time being, I’ve changed ‘while ($i <= 24)’ to ‘while ($i <= 4)’, and billing issues have cleared.

Poller health looks good. CPU utilization on server is roughly 40%. Memory utilization seems good. UI is lightning fast for the most part.