Daily.php stalls and hogs the CPU and memory

Help
1

My daily script with command as follows, is getting stuck,hogging the cpus and memory :-

    PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
 740110 librenms   20   0 4423M 4331M 23476 R 98.6 36.2 27h14:08 php /home/librenms/daily.php -f recalculate_device_dependencies
3603782 librenms   20   0 2394M 2309M 23424 R 90.4 19.3  6h22:09 php /home/librenms/daily.php -f recalculate_device_dependencies

I am getting by, by killing them daily but this is a high-maintenance solution.

The validate.php passes mostly but for advising of some files having incorrect permissions, which seem to be also changing daily (I’ve rolled the advice into a ‘daily fix’ script like :-

/usr/bin/chown -R librenms:librenms /home/librenms
/usr/bin/setfacl -d -m g::rwx /home/librenms/rrd /home/librenms/logs /home/librenms/bootstrap/cache/ /home/librenms/storage/
/usr/bin/chmod -R ug=rwX /home/librenms/rrd /home/librenms/logs /home/librenms/bootstrap/cache/ /home/librenms/storage/

(yes i do have a non-standard placement of librenms data dirs, for my /home partition is the bigger one)

I can’t help but think this might be related to each other somehow? But anyway if not, where can I find what is the ‘dependency calculation’ getting stuck on?

(I think this - the cpu/memory hog - somehow started not long after updating php to 8.1 but not immediately, i don’t remember seeing this immediately after the update.)

Thanks in advance,

1 Like
2

Hi @el_es

The daily update script does not necessarily need to force a recalculation every day.
Did you make this modification manually “-f recalculate_device_dependencies” ?
If not, you can manage the daily.sh script launch:

vi /etc/cron.d/librenms

19 0 * * * librenms /applis/librenms/daily.sh >> /dev/null 2>&1
3

I never touched any of that; this came stock when I installed (on Debian) using the official guide. it should all be stock ; as far as I can see from the daily.sh it is invoking this option

           cleanup)
                # Cleanups
                options=("refresh_alert_rules"
                               "refresh_os_cache"
                               "refresh_device_groups"
                               "recalculate_device_dependencies"
                               "syslog"
                               "eventlog"
                               "authlog"
                               "callback"
                               "device_perf"
                               "purgeusers"
                               "bill_data"
                               "alert_log"
                               "rrd_purge"
                               "ports_fdb"
                               "route"
                               "ports_purge")
                call_daily_php "${options[@]}"

i could maybe turn this off in the script but then it’ll pull again from git?

4

You sure you don’t have an infinite loop in device dependencies?

5

is there a log of this recalculation that it could show me where it is?

6

On that note, i just looked in poller_wrapper.log and…
it was alerting me to the missing memcache module for python…

2022-11-25 16:00:02,076 :: CRITICAL :: ERROR: missing memcache python module:
2022-11-25 16:00:02,076 :: CRITICAL :: On deb systems: apt-get install python3-memcache
2022-11-25 16:00:02,076 :: CRITICAL :: On other systems: pip3 install python-memcached
2022-11-25 16:00:02,077 :: CRITICAL :: Disabling distributed discovery.

it wasn’t mentioned in the validate.php output…

(on a side note, looks like the ‘wrong permissions’ error is cropping up again, i’ll come back to that

so now I installed the memcache module for python we’ll see if that does something for the hog. Though I kinda doubt it.

7

Nope, it did not help, the daily.php still stalls

 893270 librenms   20   0 4191M 3709M 13192 R 110. 31.0 30h39:58 php /home/librenms/daily.php -f recalculate_device_dependencies
2296447 librenms   20   0 4829M 4585M 12944 R 87.1 38.3 52h37:17 php /home/librenms/daily.php -f recalculate_device_dependencies
3688791 librenms   20   0 2550M 1259M 13204 R 63.7 10.5  8h47:20 php /home/librenms/daily.php -f recalculate_device_dependencies
8

The code can’t handle infinite loops. Please fix your device dependencies.

9

Thank you for your reply : i found your earlier replies to this kind of thread now (Daily script becomes a stuck process with high cpu and memory usage - #10 by murrant) and I tried to do this, but the script just prints DEBUG :

librenms@debian:~$ php /home/librenms/daily.php -f recalculate_device_dependencies -d
DEBUG

and nothing happens.

(is there any option to enable a directional arrow on the dependencies map, so that it’ll be easier to spot where the loop is happening… ?

10

answering myself :
insert this into $config['network_map_vis_options'] = '{ section (somehow by default they’re disabled)

 "edges": {
        "arrows": {
           "to": {
                 "enabled": true
           }
      }
  },

:upside_down_face:

2 Likes
11

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

12

Sounds like a good default to change upstream. :slight_smile: