Unable to delete devices. Nginx Timeout or PHP memory limit error

rokkhan · 18 August 2022 09:53

Hello,

I am trying to delete some devices but I get a 504 gateway timeout nginx error when I try by webui

and If i try by delhost.php I get this error:

PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 8388616 bytes) in /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasRelationships.php on line 632

In HasRelationships.php line 632:

Allowed memory size of 536870912 bytes exhausted (tried to allocate 8388616 bytes)

Php memory limit was sert to 128MB by default, I never changed this value but I am still getting this error changing this value to 512MB.

I have updated to the latest version and still getting this error.

–Validate Output:

./validate.php

Component	Version
LibreNMS	22.7.0-68-ga40e1d5
DB Schema	2022_07_19_081224_plugins_unique_index (244)
PHP	7.3.33
Python	3.6.8
Database	MariaDB 10.5.13-MariaDB
RRDTool	1.4.8
SNMP	5.7.2
====================================

[OK] Composer Version: 2.3.7
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK]
[OK] Database schema correct
[OK] MySQl and PHP time match
[OK] Active pollers found
[OK] Dispatcher Service not detected
[OK] Locks are functional
[OK] Python poller wrapper is polling
[OK] Redis is unavailable
[OK] rrdtool version ok
[OK] Connected to rrdcached

Any clue? Could be related to some data missing in the database about this devices?
Does anyone knows how to remove this devices manually from the database?

Thanks in advance.

murrant · 18 August 2022 15:29

Something with that particular device. I would temporarily bump the memory up to 2G or something and see if it works then.

rokkhan · 18 August 2022 21:17

it looks that it gets stucked for hours without any output

/opt/librenms/delhost.php ssw27240006.domain.es

top
top - 22:31:08 up 1 day, 3:00, 1 user, load average: 7.71, 7.88, 7.97
Tasks: 129 total, 6 running, 123 sleeping, 0 stopped, 0 zombie
%Cpu(s): 50.9 us, 24.1 sy, 0.0 ni, 23.2 id, 0.9 wa, 0.0 hi, 0.9 si, 0.0 st
KiB Mem : 12137132 total, 269944 free, 5854252 used, 6012936 buff/cache
KiB Swap: 1048572 total, 1046524 free, 2048 used. 5919572 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16052 librenms 20 0 2126368 1.6g 11632 R 96.0 14.2 239:12.98 php
27354 librenms 20 0 880804 424412 12300 S 10.6 3.5 9:32.33 php
10973 librenms 20 0 428952 41556 11960 R 9.6 0.3 0:00.29 php
11137 librenms 20 0 428696 40612 11776 S 9.3 0.3 0:00.28 php
11084 librenms 20 0 428696 40604 11776 S 8.6 0.3 0:00.26 php

I have deleted other devices without any problem but there are about 7 devices that i can not remove.

Is there any option to delete them directly on database?

Jellyfrog · 18 August 2022 22:57

I’m guessing there is a circular dependency issue.
Something points to each other and it tries to resolve it forever

murrant · 19 August 2022 01:35

Could be it, remove all device dependencies first.

RR1 · 19 August 2022 04:22

I had similar issue in my environment with 2000+ devices
i used a dirty fix by adding below time out setting in nginx’s librenms.conf

fastcgi_read_timeout 240;

rokkhan · 19 August 2022 07:27

There are no dependencies for this devices:

rokkhan · 19 August 2022 07:28

I will try but it also fails running from command line and with php interpreter. I think this way does not use nginx, isn´t it?

rokkhan · 19 August 2022 08:37

Configured like these:

server {
listen 80;
server_name librenms.domain.es;
root /opt/librenms/html;
index index.php;
access_log /opt/librenms/logs/access_log;
error_log /opt/librenms/logs/error_log;
location / {
try_files $uri $uri/ @librenms;
}
location ~ .php {
fastcgi_param PATH_INFO $fastcgi_path_info;
include fastcgi.conf;
fastcgi_split_path_info ^(.+.php)(/.+)$;
fastcgi_pass unix:/var/run/php-fpm/php7.2-fpm.sock;
fastcgi_read_timeout 240;
}
location ~ /.ht {
deny all;
}
location @librenms {
rewrite api/v0(.*)$ /api_v0.php/$1 last;
rewrite ^(.+)$ /index.php/$1 last;
}
}

And same output:

RR1 · 19 August 2022 10:05

yes, if running from cli, does not use nginx

Jellyfrog · 19 August 2022 15:05

Try commenting this line and try again;

github.com

librenms/librenms/blob/master/app/Observers/DeviceObserver.php#L175

      
        
            
            
    $device->ports()
                    ->select(['port_id', 'device_id', 'ifIndex', 'ifName', 'ifAlias', 'ifDescr'])
                    ->chunk(100, function ($ports) {
                        foreach ($ports as $port) {
                            $port->delete();
                        }
                    });
            
            
    // handle device dependency updates
                $device->children->each->updateMaxDepth($device->device_id);
            }
            
            
/**
             * Handle the device "Pivot Attached" event.
             *
             * @param  \App\Models\Device  $device
             * @param  string  $relationName  parents or children
             * @param  array  $pivotIds  list of pivot ids
             * @param  array  $pivotIdsAttributes  additional pivot attributes
             * @return void

rokkhan · 22 August 2022 15:51

it worked!!!

[librenms@-------- Observers]$ /opt/librenms/delhost.php ssw27240001.domain.es
Removed device ssw27240001.domain.es

Do i have to run anythinng else? or just uncoment once all the problematic devices have been removed…

Thanks for all

system · 20 November 2022 15:51

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.