OS update broke LibreNMS pooler

Help
1

Hi guys we updated ubuntu and it seems like its breaking:

[2020-06-18 10:56:25] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,SYSTEMD,daemon,info,info,1e,Stopping MariaDB 10.1.44 database server…,2020-06-18 10:56:25)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,SYSTEMD,daemon,info,info,1e,Stopping MariaDB 10.1.44 database server…,2020-06-18 10:56:25))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:25] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:25 139890362631936 [Note] /usr/sbin/mysqld: Normal shutdown,2020-06-18 10:56:25)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:25 139890362631936 [Note] /usr/sbin/mysqld: Normal shutdown,2020-06-18 10:56:25))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:25] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:25 139890362631936 [Note] Event Scheduler: Purging the queue. 0 events,2020-06-18 10:56:25)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:25 139890362631936 [Note] Event Scheduler: Purging the queue. 0 events,2020-06-18 10:56:25))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:25] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:25 139889525888768 [Note] InnoDB: FTS optimize thread exiting.,2020-06-18 10:56:25)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:25 139889525888768 [Note] InnoDB: FTS optimize thread exiting.,2020-06-18 10:56:25))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:25] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:25 139890362631936 [Note] InnoDB: Starting shutdown…,2020-06-18 10:56:25)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:25 139890362631936 [Note] InnoDB: Starting shutdown…,2020-06-18 10:56:25))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:26] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:26 139890362631936 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool,2020-06-18 10:56:26)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:26 139890362631936 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool,2020-06-18 10:56:26))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:28] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:27 139890362631936 [Note] InnoDB: Shutdown completed; log sequence number 117527958725,2020-06-18 10:56:27)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:27 139890362631936 [Note] InnoDB: Shutdown completed; log sequence number 117527958725,2020-06-18 10:56:27))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:28] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:27 139890362631936 [Note] /usr/sbin/mysqld: Shutdown complete,2020-06-18 10:56:27)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:27 139890362631936 [Note] /usr/sbin/mysqld: Shutdown complete,2020-06-18 10:56:27))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:28] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,SYSTEMD,daemon,info,info,1e,Stopped MariaDB 10.1.44 database server.,2020-06-18 10:56:27)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,SYSTEMD,daemon,info,info,1e,Stopped MariaDB 10.1.44 database server.,2020-06-18 10:56:27))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:30] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,SYSTEMD,daemon,info,info,1e,Starting MariaDB 10.1.44 database server…,2020-06-18 10:56:30)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,SYSTEMD,daemon,info,info,1e,Starting MariaDB 10.1.44 database server…,2020-06-18 10:56:30))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:31] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] /usr/sbin/mysqld (mysqld 10.1.44-MariaDB-0ubuntu0.18.04.1) starting as process 32026 …,2020-06-18 10:56:31)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] /usr/sbin/mysqld (mysqld 10.1.44-MariaDB-0ubuntu0.18.04.1) starting as process 32026 …,2020-06-18 10:56:31))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:31] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Using mutexes to ref count buffer pool pages,2020-06-18 10:56:31)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Using mutexes to ref count buffer pool pages,2020-06-18 10:56:31))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:31] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: The InnoDB memory heap is disabled,2020-06-18 10:56:31)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: The InnoDB memory heap is disabled,2020-06-18 10:56:31))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:31] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins,2020-06-18 10:56:31)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins,2020-06-18 10:56:31))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:31] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier,2020-06-18 10:56:31)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier,2020-06-18 10:56:31))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:31] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Compressed tables use zlib 1.2.11,2020-06-18 10:56:31)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Compressed tables use zlib 1.2.11,2020-06-18 10:56:31))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:31] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Using Linux native AIO,2020-06-18 10:56:31)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Using Linux native AIO,2020-06-18 10:56:31))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:31] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Using SSE crc32 instructions,2020-06-18 10:56:31)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Using SSE crc32 instructions,2020-06-18 10:56:31))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:31] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Initializing buffer pool, size = 307.0M,2020-06-18 10:56:31)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Initializing buffer pool, size = 307.0M,2020-06-18 10:56:31))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:31] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Completed initialization of buffer pool,2020-06-18 10:56:31)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Completed initialization of buffer pool,2020-06-18 10:56:31))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:31] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Highest supported file format is Barracuda.,2020-06-18 10:56:31)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Highest supported file format is Barracuda.,2020-06-18 10:56:31))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:31] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: 128 rollback segment(s) are active.,2020-06-18 10:56:31)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: 128 rollback segment(s) are active.,2020-06-18 10:56:31))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:31] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Waiting for purge to start,2020-06-18 10:56:31)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Waiting for purge to start,2020-06-18 10:56:31))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
[2020-06-18 10:56:31] production.ERROR: Could not connect to database! SQLSTATE[HY000] [2002] No such file or directory (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.46-86.2 started; log sequence number 117527958725,2020-06-18 10:56:31)) (SQL: INSERT IGNORE INTO syslog (device_id,program,facility,priority,level,tag,msg,timestamp) VALUES (1,MYSQLD,daemon,info,info,1e,2020-06-18 10:56:31 139717703834752 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.46-86.2 started; log sequence number 117527958725,2020-06-18 10:56:31))#0 /opt/librenms/includes/syslog.php(167): dbInsert(Array, ‘syslog’)
#1 /opt/librenms/syslog.php(24): process_syslog(Array, 1)
#2 {main}
/opt/librenms/discovery.php new 2020-06-18 11:00:01 - 0 devices discovered in 0.002 secs
/opt/librenms/discovery.php new 2020-06-18 11:05:01 - 0 devices discovered in 0.003 secs
/opt/librenms/discovery.php new 2020-06-18 11:10:02 - 0 devices discovered in 0.003 secs
/opt/librenms/discovery.php new 2020-06-18 12:20:01 - 0 devices discovered in 0.003 secs

I’m able to connect to the database with user/pass and its working… but when I check:

root@librenms:/opt/librenms# /opt/librenms/cronic /opt/librenms/discovery-wrapper.py
Cronic detected failure or error output for the command:
/opt/librenms/discovery-wrapper.py

RESULT CODE: 2

ERROR OUTPUT:

STANDARD OUTPUT:
ERROR: Could not connect to MySQL database!
ERROR: (1130, “Host ‘::1’ is not allowed to connect to this MariaDB server”)

2

As you can see from your last line, Python is trying to use ::1 as localhost so you need to grant access to 'librenms'@'::1' or configure db_socket

1 Like
3

Indeed the Python3 mysql library seems to be more strict about connection settings, so people with wonky setups don’t work anymore.

Often it is when the mysql server is not listening on localhost.