High CPU load after upgrade

[Hans_Oele]

When we switch to php 8.1 on Ubuntu 22.02 or Debian 11 we get a really high cpu load.
As soon as we switch back to php 7.1 the issue is resolved.

All updates are installed and we have configured opcache and we have run the ./daily.sh as well
Both environments are using apache2 and mod_php.

This is the output of top: CPU - LibreNMS

This is output of ./validate.php

librenms@mgt02:~$ ./validate.php

Component	Version
LibreNMS	22.9.0-8-ga174268a3
DB Schema	2022_09_03_091314_update_ports_adsl_table_with_defaults (246)
PHP	8.1.2
Python	3.10.6
Database	MariaDB 10.6.7-MariaDB-2ubuntu1.1
RRDTool	1.7.2
SNMP	5.9.1

====================================

[OK] Composer Version: 2.4.2
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK]
[OK] Database schema correct
[OK] MySQl and PHP time match
[OK] Active pollers found
[OK] Dispatcher Service not detected
[OK] Locks are functional
[OK] Python poller wrapper is polling
[OK] Redis is unavailable
[OK] rrd_dir is writable
[OK] rrdtool version ok
librenms@mgt02:~$

[murrant]

What was causing the CPU load?

Is this a particular device? All devices? (The poller or discovery, or other) What does the debug output look like for the affected item? Does it appear to hang in a specific module?

[Hans_Oele]

This is fo all devices.
I do have to say that on the ubuntu environment we have 760 devices on 1 server.

[murrant]

Any errors in the log? My CPU usage is lower after upgrading 45%->40%. 600 devices on one server.

[murrant]

Hmm, it is actually down to about 10% (from 45%) on average after settling down.

[murrant]

I found an issue with the version code being called many times and running some heavy operations.

Cache version data

Can you try again now?

[Hans_Oele]

I’ve run ./daily.sh again.
unfortunately this still doesn’t resolves the issue.

librenms@mgt02:~$ ./validate.php

Component	Version
LibreNMS	22.9.0-14-gf60b6788d
DB Schema	2022_09_03_091314_update_ports_adsl_table_with_defaults (246)
PHP	8.1.2
Python	3.10.6
Database	MariaDB 10.6.7-MariaDB-2ubuntu1.1
RRDTool	1.7.2
SNMP	5.9.1
====================================

[OK] Composer Version: 2.4.2
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK]
[OK] Database schema correct
[OK] MySQl and PHP time match
[OK] Active pollers found
[OK] Dispatcher Service not detected
[OK] Locks are functional
[OK] Python poller wrapper is polling
[OK] Redis is unavailable
[OK] rrd_dir is writable
[OK] rrdtool version ok
[WARN] Your local git contains modified files, this could prevent automatic updates.
[FIX]:
You can fix this with ./scripts/github-remove
Modified Files:
rrd/.gitignore
librenms@mgt02:~$

[murrant]

Maybe you are having a different issue? Short of you installing xdebug and profiling the poller, I’m not sure how you can find out what it is.

[Andy_N]

I appear to be getting a similar issue with high cpu since last nights daily script. However I’m on monthly release, still using php 7.4 and Ubuntu 20.04, so not the same. but consist high cpu. No obvious reason as far as i can tell.

[murrant]

Why do you have multiple librenms-service.py processes? AFAIK, you should only have one.

[Andy_N]

Yeah, glad you think that too cos definitely not what I expected either. I was seeing 2-6 discovery.php processes start every second! We have about 900 devices so I wouldn’t expect that many! Most likely the reason my CPU was maxing out though. I didn’t upgrade to php8.1 or Ubuntu 22, and we’re on monthly release so no code changes that I know of that night! What’s more it all settled after 38 hours with no intervention. And to top it off, CPU now running at 45% average rather than 70% previously. It doesn’t really make sense! Oh and my daily.sh log file has no entries for the time it went weird.

Only common feature with @Hans_Oele is we’re both running Ubutnu so maybe it’s Ubuntu issue or python on Ubuntu issue. Not sure how to find out retrospectively.

@murrant what do you put the spike in traffic on you graph down too?