Daily script becomes a stuck process with high cpu and memory usage

Aguilo_Network · 5 November 2020 10:08

Hello all,
i’ve suddenly encountered an issue on my librenms installation.
Installation details: i5 750, 12GB of ram, Ubuntu server 18.04. apache2 (physical installation, not a VM)
24 devices polled by one poller, default configuration of librenms. no change done on cron for librenms. The only config done are : mail transport, custome alert rules, devices/services/app definition, dependencies and dashboard.
as a new forum user i cannot post more than one screenshot, so i’ve put it via links.

yesterday, without having done any change on my librenms, i started to received temperature alerts for my librenms server itself. After a check i’ve discovered that there is a high cpu usage and memory usage. htop gave me the information that it is due to php called by librenms with script :

“daily.php -f recalculate_device_dependencies”

i then restarted apache, updaded the system with apt-get update && apt-get upgrade, ran daily.sh update, rebooted the server. after a reboot issue was gone, but as soon as the cron run for daily.sh at 00:15 am, issue came back. the process is still active at 11.00am like it was stuck in a kind of loop of dependencies (based on the name of the command). Then i’ve done a validate.php, except python in the GUI warning nothing came out.

The GUI is responsive, the poller performance is correct and stable.
i tried to find an explanation on google, but no result with the command as search.

If somebody has any idea, thank you for you much appreciated help.

In the below link is my devices dependencies map in which you can see that there is no loop.
you will also find additionnal capture on this imgbb album including cpu/mem usage and temperature : captures — ImgBB

Here is my output of validate.php

Component	Version
LibreNMS	1.69-5-g240edd214
DB Schema	2020_07_27_00522_alter_devices_snmp_algo_columns (188)
PHP	7.4.11
Python	3.6.9
MySQL	10.1.47-MariaDB-0ubuntu0.18.04.1
RRDTool	1.7.0
SNMP	NET-SNMP 5.7.3
OpenSSL
====================================

[OK] Composer Version: 1.10.17
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct

have a great day

Aguilo_Network · 6 November 2020 08:27

Hello again, i add that the logs of daily script are missing some timestamp, but i can see that the day before the first incident, or maybe the same day, an update has been downloaded.
below the logs of daily.

thank you

Fetching notifications
[ Tue, 03 Nov 2020 00:46:34 +0100 ] LibreNMS Notifications (48)
[ Tue, 03 Nov 2020 00:46:34 +0100 ] misc/notifications.rss (60)
[ Tue, 03 Nov 2020 00:46:34 +0100 ] Updating DB Done
Returned: 0
Caching PeeringDB data
Peering DB integration disabled
Returned: 0
Updating to latest codebase

Returned: 0
Updating Composer packages

LibreNMS\ComposerHelper::preInstall
Loading composer repositories with package information
Installing dependencies from lock file
Package operations: 4 installs, 43 updates, 3 removals

Removing symfony/polyfill-php70 (v1.18.1)
Removing paragonie/random_compat (v9.99.99)
Removing symfony/dotenv (v3.4.45)
Downgrading amenadiel/jpgraph (3.6.21 => 3.6.14): As there is no ‘unzip’ command installed zip files are being unpacked using the PHP zip extension.
This may cause invalid reports of corrupted archives. Besides, any UNIX permissions (e.g. executable) defined in the archives will be lost.
Installing ‘unzip’ may remediate them.
Downloading (100%)
Updating symfony/polyfill-php80 (v1.18.1 => v1.20.0): Downloading (100%)
Updating symfony/polyfill-mbstring (v1.18.1 => v1.20.0): Downloading (100%)
Updating symfony/polyfill-ctype (v1.18.1 => v1.20.0): Downloading (100%)
Installing graham-campbell/result-type (v1.0.1): Downloading (100%)
Updating vlucas/phpdotenv (v4.1.8 => v5.2.0): Downloading (100%)
Updating symfony/css-selector (v5.1.6 => v5.1.7): Loading from cache
Updating symfony/var-dumper (v5.1.6 => v5.1.7): Loading from cache
Updating symfony/routing (v5.1.6 => v5.1.7): Downloading (100%)
Updating symfony/process (v5.1.6 => v5.1.7): Loading from cache
Updating symfony/polyfill-php72 (v1.18.1 => v1.20.0): Downloading (100%)
Updating symfony/polyfill-intl-normalizer (v1.18.1 => v1.20.0): Downloading (100%)
Updating symfony/polyfill-intl-idn (v1.18.1 => v1.20.0): Downloading (100%)
Updating symfony/mime (v5.1.6 => v5.1.7): Loading from cache
Updating symfony/polyfill-php73 (v1.18.1 => v1.20.0): Downloading (100%)
Updating symfony/http-foundation (v5.1.6 => v5.1.7): Downloading (100%)
Updating symfony/http-client-contracts (v2.2.0 => v2.3.1): Downloading (100%)
Updating symfony/event-dispatcher (v5.1.6 => v5.1.7): Loading from cache
Updating symfony/error-handler (v5.1.6 => v5.1.7): Downloading (100%)
Updating symfony/http-kernel (v5.1.6 => v5.1.7): Downloading (100%)
Updating symfony/finder (v5.1.6 => v5.1.7): Loading from cache
Installing symfony/polyfill-intl-grapheme (v1.20.0): Downloading (100%)
Installing symfony/string (v5.1.7): Downloading (100%)
Updating symfony/console (v5.0.11 => v5.1.7): Downloading (100%)
Updating symfony/polyfill-iconv (v1.18.1 => v1.20.0): Downloading (100%)
Updating opis/closure (3.5.7 => 3.6.0): Downloading (100%)
Updating symfony/translation-contracts (v2.2.0 => v2.3.0): Downloading (100%)
Updating symfony/translation (v5.1.6 => v5.1.7): Loading from cache
Updating nesbot/carbon (2.40.1 => 2.41.5): Downloading (100%)
Updating league/mime-type-detection (1.5.0 => 1.5.1): Downloading (100%)
Updating league/commonmark (1.5.5 => 1.5.6): Downloading (100%)
Updating dragonmantank/cron-expression (v2.3.0 => v3.0.2): Downloading (100%)
Updating laravel/framework (v7.28.3 => v8.11.2): Downloading (100%)
Removing darkghosthunter/larapoke (v4.1.1)
Installing darkghosthunter/larapoke (dev-master 2a58fc8): Downloading (100%)
Updating doctrine/dbal (2.10.4 => 2.12.0): Downloading (100%)
Downgrading fico7489/laravel-pivot (3.0.6 => 3.0.3): Downloading (100%)
Updating fideloper/proxy (4.4.0 => 4.4.1): Downloading (100%)
Updating fruitcake/laravel-cors (v2.0.2 => v2.0.3): Downloading (100%)
Installing psr/http-client (1.0.1): Downloading (100%)
Updating guzzlehttp/psr7 (1.6.1 => 1.7.0): Downloading (100%)
Updating guzzlehttp/promises (v1.3.1 => 1.4.0): Downloading (100%)
Updating guzzlehttp/guzzle (6.5.5 => 7.2.0): Downloading (100%)
Updating laravel/ui (v2.4.1 => v3.0.0): Downloading (100%)
Updating phpmailer/phpmailer (v6.1.7 => v6.1.8): Downloading (100%)
Updating symfony/yaml (v4.4.14 => v4.4.15): Loading from cache
Updating tightenco/ziggy (v0.8.1 => 0.9.4): Downloading (100%)
Updating wpb/string-blade-compiler (dev-laravel-7-and-autoload-blade-custom-directives f4666b7 => 6.0.2): Downloading (100%)
Generating optimized autoload files
Deprecation Notice: Class Crypt_CHAP_MD5 located in ./vendor/dapphp/radius/lib/Pear_CHAP.php does not comply with psr-0 autoloading standard. It will not autoload anymore in Composer v2.0. in phar:///opt/librenms/composer.phar/src/Composer/Autoload/ClassMapGenerator.php:201
Deprecation Notice: Class Crypt_CHAP_MSv1 located in ./vendor/dapphp/radius/lib/Pear_CHAP.php does not comply with psr-0 autoloading standard. It will not autoload anymore in Composer v2.0. in phar:///opt/librenms/composer.phar/src/Composer/Autoload/ClassMapGenerator.php:201
Deprecation Notice: Class Crypt_CHAP_MSv2 located in ./vendor/dapphp/radius/lib/Pear_CHAP.php does not comply with psr-0 autoloading standard. It will not autoload anymore in Composer v2.0. in phar:///opt/librenms/composer.phar/src/Composer/Autoload/ClassMapGenerator.php:201

Illuminate\Foundation\ComposerScripts::postAutoloadDump
@php artisan package:discover --ansi
Discovered Package: darkghosthunter/larapoke
Discovered Package: fideloper/proxy
Discovered Package: fruitcake/laravel-cors
Discovered Package: laravel/tinker
Discovered Package: laravel/ui
Discovered Package: librenms/laravel-vue-i18n-generator
Discovered Package: nesbot/carbon
Discovered Package: oriceon/toastr-5-laravel
Discovered Package: tightenco/ziggy
Discovered Package: wpb/string-blade-compiler
Package manifest generated successfully.
52 packages you are using are looking for funding.
Use the composer fund command to find out more!
LibreNMS\ComposerHelper::postInstall
Illuminate\Foundation\ComposerScripts::postInstall
@php artisan vue-i18n:generate --multi-locales --format=umd
@php artisan view:cache
Compiled views cleared!
Blade templates cached successfully!
@php artisan optimize
Configuration cache cleared!
Configuration cached successfully!
Route cache cleared!
Routes cached successfully!
Compiled views cleared!
Blade templates cached successfully!
Files cached successfully!
@php artisan config:clear
Configuration cache cleared!
scripts/check_requirements.py || pip3 install --user -r requirements.txt || :
Returned: 0
Updated from 0ac05fda6 to 28591c033

Returned: 0
Updating SQL-Schema
Nothing to migrate.
Returned: 0
Updating submodules

Returned: 0
Cleaning up DB
Updating to latest codebase

Returned: 0
Updating Composer packages

LibreNMS\ComposerHelper::preInstall
Loading composer repositories with package information
Installing dependencies from lock file
Nothing to install or update
Generating optimized autoload files
Deprecation Notice: Class Crypt_CHAP_MD5 located in ./vendor/dapphp/radius/lib/Pear_CHAP.php does not comply with psr-0 autoloading standard. It will not autoload anymore in Composer v2.0. in phar:///opt/librenms/composer.phar/src/Composer/Autoload/ClassMapGenerator.php:201
Deprecation Notice: Class Crypt_CHAP_MSv1 located in ./vendor/dapphp/radius/lib/Pear_CHAP.php does not comply with psr-0 autoloading standard. It will not autoload anymore in Composer v2.0. in phar:///opt/librenms/composer.phar/src/Composer/Autoload/ClassMapGenerator.php:201
Deprecation Notice: Class Crypt_CHAP_MSv2 located in ./vendor/dapphp/radius/lib/Pear_CHAP.php does not comply with psr-0 autoloading standard. It will not autoload anymore in Composer v2.0. in phar:///opt/librenms/composer.phar/src/Composer/Autoload/ClassMapGenerator.php:201
Illuminate\Foundation\ComposerScripts::postAutoloadDump
@php artisan package:discover --ansi
Discovered Package: darkghosthunter/larapoke
Discovered Package: fideloper/proxy
Discovered Package: fruitcake/laravel-cors
Discovered Package: laravel/tinker
Discovered Package: laravel/ui
Discovered Package: librenms/laravel-vue-i18n-generator
Discovered Package: nesbot/carbon
Discovered Package: oriceon/toastr-5-laravel
Discovered Package: tightenco/ziggy
Discovered Package: wpb/string-blade-compiler
Package manifest generated successfully.
52 packages you are using are looking for funding.
Use the composer fund command to find out more!
LibreNMS\ComposerHelper::postInstall
Illuminate\Foundation\ComposerScripts::postInstall
@php artisan vue-i18n:generate --multi-locales --format=umd
@php artisan view:cache
Compiled views cleared!
Blade templates cached successfully!
@php artisan optimize
Configuration cache cleared!
Configuration cached successfully!
Route cache cleared!
Routes cached successfully!
Compiled views cleared!
Blade templates cached successfully!
Files cached successfully!
@php artisan config:clear
Configuration cache cleared!
scripts/check_requirements.py || pip3 install --user -r requirements.txt || :
Returned: 0
Updated from 28591c033 to 240edd214

Returned: 0
Updating SQL-Schema
Nothing to migrate.
Returned: 0
Updating submodules

Returned: 0
Cleaning up DB
Updating to latest codebase

Returned: 0
Updating Composer packages

LibreNMS\ComposerHelper::preInstall
Loading composer repositories with package information
Installing dependencies from lock file
Nothing to install or update
Generating optimized autoload files
Deprecation Notice: Class Crypt_CHAP_MD5 located in ./vendor/dapphp/radius/lib/Pear_CHAP.php does not comply with psr-0 autoloading standard. It will not autoload anymore in Composer v2.0. in phar:///opt/librenms/composer.phar/src/Composer/Autoload/ClassMapGenerator.php:201
Deprecation Notice: Class Crypt_CHAP_MSv1 located in ./vendor/dapphp/radius/lib/Pear_CHAP.php does not comply with psr-0 autoloading standard. It will not autoload anymore in Composer v2.0. in phar:///opt/librenms/composer.phar/src/Composer/Autoload/ClassMapGenerator.php:201
Deprecation Notice: Class Crypt_CHAP_MSv2 located in ./vendor/dapphp/radius/lib/Pear_CHAP.php does not comply with psr-0 autoloading standard. It will not autoload anymore in Composer v2.0. in phar:///opt/librenms/composer.phar/src/Composer/Autoload/ClassMapGenerator.php:201
Illuminate\Foundation\ComposerScripts::postAutoloadDump
@php artisan package:discover --ansi
Discovered Package: darkghosthunter/larapoke
Discovered Package: fideloper/proxy
Discovered Package: fruitcake/laravel-cors
Discovered Package: laravel/tinker
Discovered Package: laravel/ui
Discovered Package: librenms/laravel-vue-i18n-generator
Discovered Package: nesbot/carbon
Discovered Package: oriceon/toastr-5-laravel
Discovered Package: tightenco/ziggy
Discovered Package: wpb/string-blade-compiler
Package manifest generated successfully.
52 packages you are using are looking for funding.
Use the composer fund command to find out more!
LibreNMS\ComposerHelper::postInstall
Illuminate\Foundation\ComposerScripts::postInstall
@php artisan vue-i18n:generate --multi-locales --format=umd
@php artisan view:cache
Compiled views cleared!
Blade templates cached successfully!
@php artisan optimize
Configuration cache cleared!
Configuration cached successfully!
Route cache cleared!
Routes cached successfully!
Compiled views cleared!
Blade templates cached successfully!
Files cached successfully!
@php artisan config:clear
Configuration cache cleared!
scripts/check_requirements.py || pip3 install --user -r requirements.txt || :
Returned: 0
Updating SQL-Schema
Nothing to migrate.
Returned: 0
Updating submodules

Returned: 0
Cleaning up DB
Refreshing alert rules queries
Clearing OS cache
Refreshing device group table relationships
Syslog cleared for entries over 30 days (about 0 rows)
Eventlog cleared for entries over 30 days
Perf times cleared for entries over 30 days
Device perf cleared for entries over 7 days
Deleting alert_logs more than 365 days that are not active
Deleting history of active alert_logs more than 365 days
Returned: 0

Aguilo_Network · 8 November 2020 04:37

hello all,
i’ve removed my dependencies, and problem disapeared.
so it seems there is a bug in the last version for dependencies calculation.

Dejan · 1 December 2020 21:53

I have this issue as well. Are you talking about device dependencies?

robvas · 2 December 2020 16:53

Nevermind!

Aguilo_Network · 2 December 2020 17:34

Hello guys, no i was talking about the device dependencies as indicated on the script command. So i’ve tried to remove my device dependencies, and the problem disapeared. However, as my installation is small, i did not try to recreate my dependencies to see if the issue was coming back.

Dejan · 2 December 2020 21:03

Ok. I thought so
I did the same thing but i also had to kill process as well. I haven’t tried to recreate dependencies. It’s not a big deal to me but later on i will give it a shot. I’m hoping someone from developer team will have some input.

murrant · 3 December 2020 01:18

Device dependencies? There has not been any changes to that code in some time. Perhaps a circular dependency?

Aguilo_Network · 3 December 2020 05:41

let us know if the issue persist after the cron job start it again. i also had to kill the process, but the daily script reproduced the issue each night until i remove my device dependencies.

As mentionned on my initial post, no change have been done by me, it was working perfectly until this issue occurs, probably after an update. No circular dependencies. i’ve put my dependencies map on the first post. It is may be a side effect of another change.

BR

murrant · 3 December 2020 14:09

Can you run recalculate_device_dependencies and monitor the sql queries so we see what it is running? (enable the general_log in mysqld) Or maybe just adding -d to the command might work.

Bruno-adi · 6 May 2021 15:09

Hy,

Same here.

LibreNMS  | 21.4.0-64-g76910e418
DB Schema | 2021_04_08_151101_add_foreign_keys_to_port_group_port_table (208)
PHP       | 7.3.27-1~deb10u1
Python    | 3.7.3
MySQL     | 10.3.27-MariaDB-0+deb10u1
RRDTool   | 1.7.1
SNMP      | NET-SNMP 5.7.3
====================================

[OK]    Composer Version: 2.0.13
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct

Is there a fix or a workaround ?

Aguilo_Network · 7 May 2021 04:15

Hello.
As i was moving, i did not have the Time to troubleshoot the SQL queries as suggested by Murrant. However, as said before, i’ve removed my devices dependencies and the problem disapeared. If it is not a big deal for you to remove it, it is the workaround which worked for me. I’ve recreated it some weeks after and the issue did not come back.
BR

Bruno-adi · 12 May 2021 09:12

Thanks for your answer.

I removed only mutliple dependencies from one device and the problem also disapeared.
Keeping single dependencies works for me

murrant · 14 May 2021 13:25

This means you probably created a dependency loop.

Aguilo_Network · 14 May 2021 18:30

If you look at my screenshots, i did not have any loop, but removing my dependencies solved the issue. I cannot Say for Bruno. And i Hope the code is able to detect a loop, as the device ID is unique, it is quite basic to detect an ID already seen…

murrant · 15 May 2021 02:30

Not sure the code can detect distant loops. Any code improvements are welcome, as LibreNMS is a community project.

system · 15 May 2023 02:30

This topic was automatically closed 730 days after the last reply. New replies are no longer allowed.