production.ERROR ... functions.inc.php

Help
1

Hi,

since around wednesday, 2019-06-05 I have problems with the alert notifications interface: …/alerts

When i open that URL, the interface says “Loading…”, later “No results found!”
and in ~/logs/librenms.logs I can see

[2019-06-11 08:33:36] production.ERROR: Maximum execution time of 30 seconds exceeded {“userId”:4,“email”:“[email protected]”,“exception”:"[object] (Symfony\Component\Debug\Exception\FatalErrorException(code: 1): Maximum execution time of 30 seconds exceeded at /opt/librenms/includes/html/functions.inc.php:1057)
[stacktrace]
#0 {main}
"}

If I do a API-call on alerts:
https://127.0.0.1/api/v0/alerts
I do see a count of 16…

I noticed this problem last wednesday… ./daily did not work on that night before.
So I ran it manually in the morning, that’s when the problem started.

At the same time I did on OS-update… so I thought it might be related to this…
That’s why I went back to my VM snpashot, reverting the updates of daily.sh and also the OS updates.

On the thursday night daily.sh ran again… bringing back this problem.

Please help to identify the core of the problem.

====================================

Component Version
LibreNMS 1.52-39-g1ae77df
DB Schema 2019_05_12_202408_create_mpls_lsp_paths_table (134)
PHP 7.1.29
MySQL 5.5.60-MariaDB
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2
====================================

[OK] Composer Version: 1.8.5
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct

Thank you!

2

So from that error would hint the script is timing out from this rtrim function

            $fault_detail = rtrim($fault_detail, ', ');

Try set max_execution_time = 360 in your php.ini then restart the webserver. Go to alert notifications again and wait 6 minutes, if still hasn’t loaded then it would seem to be an endless loop.

1 Like
3

That doesn’t make much sense at all. Perhaps if you posted the full stacktrace instead of part of it?

4

Hi Murant,
thank you for your reply.

Well, actually, that was the full stacktrace I could see in librenms.log.
Digging a lil deeper I found a longer stracktrace:

[stacktrace]
#0 {main}
"}
40
8
8
8
37
37
54
7
8
54
40
8
8
28
8
40
28
8
37
37
54
37
7
7
8
8
37
54
54
8
54
28
40
28
28
8
37
7
7
8
37
55
55
55
54
33
8
28
54
55
61
61
28
55
7
33
8
7
8
8
55
55
41
55
54
33
8
28
55
55
55
55
55
61
55
8
2
41
61
55
7
55
55
33
55
8
8
41
54
55
55
55
55
55
8
2
41
55
55
55

I also found something (more) interesting:

[2019-06-12 07:47:16] production.ERROR: Allowed memory size of 536870912 bytes exhausted (tried to allocate 4096 bytes) {“exception”:"[object] (Symfony\Component\Debug\Exception\FatalErrorException(code: 1): Allowed memory size of 536870912 bytes exhausted (tried to allocate 4096 bytes) at /opt/librenms/includes/alerts.inc.php:217)
[stacktrace]
#0 {main}
"}
61
8
61
8
8
8
40
8
40
8
8
8
8
8
37
37
37
37
54
54
7
8
7
54
8
54
8
28
40
28
8
37
28
7
40
28
8
37
37
54
7
8
54
37
54
28
54
8
7
28
55
8
55
55
33
7
8
28
55
55
55
33
8
28
55
61
55
61
61
61
55
7
33
55
7
8
33
8
8
41
54
8
41
54
55
55
55
55
55
8
55
55
55
55
55
8
2
41
2
41

… so I set
memory_limit = 512M
in php.ini (increasing from 128M) and rebooted.

The problem however still exists…

5

The above error already said 512M exhausted. Also, that is a lot of RAM, I wonder why you are causing that.

6

I believe I have two different issues:

Maximum execution time of 30 seconds exceeded at /opt/librenms/includes/html/functions.inc.php:1057)

seems to be related to low DB performance. Now that I reduced load on it (syslogging), the behavior of /alerts returned to normal.

Allowed memory size of 536870912 bytes exhausted (tried to allocate 4096 bytes) {“exception”:"[object] (Symfony\Component\Debug\Exception\FatalErrorException(code: 1): Allowed memory size of 536870912 bytes exhausted (tried to allocate 4096 bytes) at /opt/librenms/includes/alerts.inc.php:217)

hasn’t appeared in librenms.log any more.

There were 5 other production errors:

production.ERROR: Allowed memory size of 536870912 bytes exhausted (tried to allocate 8388616 bytes) {“exception”:"[object] (Symfony\Component\Debug\Exception\FatalErrorException(code: 1): Allowed memory size of 536870912 bytes exhausted (tried to allocate 8388616 bytes) at /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasRelationships.php:554)

… et I don’t see any impact through them.

7

Actually I’ve seen this before. When have a alert rule that checks a log without the past_5m macro, includes ALL log entries in the alert details.

Go fix your alert rules.