Fastly increasing librenms.log file

Help
1

Hi there,

Since sometimes last week, I think Sunday the log file started to increase rapidly(it got to 97gb) and it filled up the hard drive. Initially I thought the issue was with one of the DC sending too much data. But after resolving that and checked on the log file after a couple of days it has grown up to 17gb.
Here it is the validation.php

Component Version
LibreNMS 21.10.2
DB Schema 2021_10_03_164200_update_hrsystem_table (221)
PHP 7.4.6
Python 3.6.8
MySQL 10.5.12-MariaDB
RRDTool 1.4.8
SNMP NET-SNMP 5.7.2

====================================

[OK] Composer Version: 2.1.10
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct

I ran daily.sh with no issues either.

These are the errors I see in the log file.
production.ERROR: PDOException: SQLSTATE[HY000] [2002] Permission denied in /opt/librenms/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:40
Next Doctrine\DBAL\Driver\PDO\Exception: SQLSTATE[HY000] [2002] Permission denied in /opt/librenms/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDO/Exception.php:18
Next Illuminate\Database\QueryException: SQLSTATE[HY000] [2002] Permission denied (SQL: select * from config where (config_name = rrdtool_version) limit 1) in /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Connection.php:703

I can connect just fine to the database, I also ran mysqlcheck librenms -p --auto-repair
in case something got broken. But still the problem persists

Thanks,
Mladen

2

Librenms can’t connect to the database with your credentials you provided. Adjust .env accordingly

3

I haven’t changed anything in .env file, and i can confirm the user pass and DB name are correct. I saw in another topic user changed db_host from localhost to 127.0.0.1 but still, the problem persists. I also tried to add single and double quotes around the password. Since nothing really worked I restored the file to default.

note: also if any of the DB credentials are wrong, it won’t pass validate test. I just played some more with quotes and made sure there is no extra space anywhere, even though as I said earlier I haven’t touch .env file before the problem started.

4

Credentials may be OK - it looks more like the user credentials you are connecting with don’t have permissions to access some of the tables.

" Permission denied (SQL: select * from config where (config_name = rrdtool_version) limit 1)"

Double check access rights (grants) for the SQL user LibreNMS is using to all the tables.

5

I did the following to grant the librenms user all privileges
GRANT ALL PRIVILEGES ON librenms.* TO ‘librenms’@‘localhost’;
FLUSH PRIVILEGES;
That didn’t help cuz the errors were still coming in so I granted the user access to all the dbs, but still the problem persists and I see the errors.
GRANT ALL PRIVILEGES ON . TO ‘librenms’@‘localhost’;
FLUSH PRIVILEGES;

After this I feel like the log file is increasing its size way faster than before.

6

Login with the credentials you have in the .env file and run select * from config where (config_name = rrdtool_version) limit 1; does it work for you?

7

This is what im getting when I ran
select * from config where (config_name = rrdtool_version) limit 1;

ERROR 1054 (42S22): Unknown column ‘rrdtool_version’ in ‘where clause’

8

I wonder if quotes are being stripped in the logs, try this:

select * from `config` where (`config_name` = 'rrdtool_version') limit 1;
9

I think they are, cuz i got this after you fixed the quotes

MariaDB [librenms]> select * from config where (config_name = ‘rrdtool_version’) limit 1;
±----------±----------------±-------------+
| config_id | config_name | config_value |
±----------±----------------±-------------+
| 33 | rrdtool_version | “1.4.8” |
±----------±----------------±-------------+
1 row in set (0.001 sec)

10

Is there anything else that I can try to solve this problem? Right now I have added a bash script to remove files bigger than 500mb in the logs folder as a cron job to mitigate the problem, but this is definitely not a solution.

11

Is LibreNMS working? Because if you have that error, it should not be. (If it is working, where is the error coming from?)

try this lnms config:get rrdtool_version

and this:
lnms tinker --execute="dump(\App\Models\Config::where('config_name', 'rrdtool_version')->first()->toArray())"

Also, is that the complete error stacktrace? becuase I don’t see any LibreNMS files listed (only vendor files).

12

It looks like its working at least from the UI I’m also getting the alarms emails.
This is what I’m getting from the above commands.

[librenms@librenms ~]$ lnms config:get rrdtool_version
1.4.8
[librenms@librenms ~]$ lnms tinker --execute=“dump(\App\Models\Config::where(‘config_name’, ‘rrdtool_version’)->first()->toArray())”
^ array:3 [
“config_id” => 33
“config_name” => “rrdtool_version”
“config_value” => “1.4.8”
]

13

I don’t know what happened since I haven’t made any changes, but it seems that the problem got resolved by itself. The only thing I did was just run the daily.sh script

14

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.