Hi All,
We are having issues with one of LibreNMS servers. We 2 Ubuntu18.04 boxes running a distributed setup.
Server 02 - DB, Web, Poller
Server 03 - Poller
Server 03’s poller is running and updating the DB fine
Server 02’s isn’t running from the cron job, This stopped working on 16th of May.
This is the ./validate from Server 02
root@02:/opt/librenms# ./validate.php -distributed
====================================
Component | Version
--------- | -------
LibreNMS | 1.63-152-ga18403584
DB Schema | 2020_04_06_001048_the_great_index_rename (166)
PHP | 7.4.6
Python | 3.6.9
MySQL | 10.1.44-MariaDB-0ubuntu0.18.04.1
RRDTool | 1.7.0
SNMP | NET-SNMP 5.7.3
====================================
[OK] Composer Version: 1.10.6
[OK] Dependencies up-to-date.
Checking configuration: OK
Checking database: OK
[OK] Database connection successful
[OK] Database schema correct
Checking disk: OK
Checking php: OK
Checking programs: OK
Checking python: OK
Checking rrd: OK
Checking updates: OK
Checking user: OK
This is the cron job on Server 02, other jobs are running OK
root@02:/etc/cron.d# cat librenms
# Using this cron file requires an additional user on your system, please see install docs.
33 */6 * * * librenms /opt/librenms/discovery-wrapper.py 8 >> /dev/null 2>&1
#33 */6 * * * librenms /opt/librenms/discovery.php -h all >> /dev/null 2>&1
*/5 * * * * librenms /opt/librenms/discovery.php -h new >> /dev/null 2>&1
*/5 * * * * librenms /opt/librenms/cronic /opt/librenms/poller-wrapper.py 32
#*/5 * * * * librenms /opt/librenms/cronic /opt/librenms/librenms-service.py 32
15 0 * * * librenms /opt/librenms/daily.sh >> /dev/null 2>&1
* * * * * librenms /opt/librenms/alerts.php >> /dev/null 2>&1
#*/5 * * * * librenms /opt/librenms/poll-billing.php >> /dev/null 2>&1
#01 * * * * librenms /opt/librenms/billing-calculate.php >> /dev/null 2>&1
#*/5 * * * * librenms /opt/librenms/check-services.php >> /dev/null 2>&1
#
*/5 * * * * librenms /opt/librenms/html/plugins/Weathermap/map-poller.php >> /dev/null 2>&1
If I run the poller manually on Server 02 it runs fine;
./poller.php host07 2020-05-27 09:29:16 - 1 devices polled in 52.85 secs
SNMP [50/29.48s]: Get[18/0.85s] Getnext[4/0.17s] Walk[28/28.47s]
MySQL [285/2.51s]: Cell[17/0.01s] Row[-16/-0.01s] Rows[29/0.06s] Column[0/0.00s]
Update[253/2.44s] Insert[2/0.01s] Delete[0/0.00s]
RRD [324/1.31s]: Other[162/1.28s] Update[162/0.03s]
If I try and run the poller wrapper as the librenms user I get the following error.
$ ./poller-wrapper.py
ERROR: missing memcache python module:
On deb systems: apt-get install python-memcache
On other systems: easy_install python-memcached
Disabling distributed poller.
ERROR: Could not connect to MySQL database!
ERROR: (1045, "Access denied for user 'librenms'@'127.0.0.1' (using password: YES)")
The modules are installed
root@02:~# apt list php-memcached python-memcache
Listing... Done
php-memcached/bionic,now 3.1.4+2.2.0-1+ubuntu18.04.1+deb.sury.org+20191129 amd64 [installed]
python-memcache/bionic-updates,bionic-updates,now 1.57-2ubuntu0.18.04.1 all [installed]
I can also connect to the DB as the librenms user using the detail is config.php
root@02:/opt/librenms# su librenms
$ mysql -h localhost -u librenms -pstrongpassword librenms
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 8023
Server version: 10.1.44-MariaDB-0ubuntu0.18.04.1 Ubuntu 18.04
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [librenms]>
I don’t know whta else could be stopping the cron job running and polling devices.
