Borked install?

Hi All,
I recently got LibreNMS up and running on my Centos 7 system (Apache)- all was running fine.
I made the move to using rrdcached, with the plan to use a couple of distributed pollers in the future, and ever since doing this, I now have the following issues:

  • Graphs will no longer draw
  • Certain pages will not load at all (for example the device list - the main page loads, and where the list loads it just says “Loading…”
  • Every page load requires the user to re-authenticate
  • Some pages cannot be viewed at all (just keeps looping back to a login screen) - for example /ports/

I have tried reverting the changes I made, but the issues persist.
This is the relevant parts of config.php:

    # rrdtool stuff
    $config['rrdtool_version'] = '1.6.0';
    $config['rrdcached'] = "localhost:42217";

    ## Poller
    $config['distributed_poller_group']          = '0';
    $config['distributed_poller_memcached_host'] = "localhost";
    $config['distributed_poller_memcached_port'] = 11211;
    $config['distributed_poller']                = true;
    $config['update']                            = 0;

This is the contents of /etc/systemd/system/rrdcached.service:

    [Unit]
    Description=Data caching daemon for rrdtool
    After=network.service

    [Service]
    Type=forking
    PIDFile=/run/rrdcached.pid
    ExecStart=/usr/sbin/rrdcached -w 1800 -z 1800 -f 3600 -s librenms -U 
librenms -G librenms -B -R -j /var/tmp -l 0.0.0.0:42217 -t 4 -F -b /opt/librenms/rrd/

    [Install]
    WantedBy=default.target

This is the contents of cat /etc/sysconfig/memcached:

    PORT="11211"
    USER="memcached"
    MAXCONN="1024"
    CACHESIZE="64"
    OPTIONS="-l 0.0.0.0 -P /var/run/memcached.pid"

Both memcached and rrdcachedd are running:

    [root@globlibrenms01 librenms]# pgrep -a memca
    1046 /usr/bin/memcached -u memcached -p 11211 -m 64 -c 1024 -l 0.0.0.0 -P /var/run/memcached.pid
    [root@globlibrenms01 librenms]# pgrep -a rrd
    1481 /usr/sbin/rrdcached -w 1800 -z 1800 -f 3600 -s librenms -U librenms -G librenms -B -R -j /var/tmp -l 0.0.0.0:42217 -t 4 -F -b /opt/librenms/rrd

and listening on their correct ports:

    [root@globlibrenms01 librenms]# netstat -an | grep 11211
    tcp        0      0 0.0.0.0:11211           0.0.0.0:*               LISTEN
    udp        0      0 0.0.0.0:11211           0.0.0.0:*
    [root@globlibrenms01 librenms]# netstat -an | grep 42217
    tcp        0      0 0.0.0.0:42217           0.0.0.0:*               LISTEN

This is the output of validate.php:

[root@globlibrenms01 librenms]# ./validate.php
====================================
Component | Version
--------- | -------
LibreNMS  | 1.41-36-g67c585b
DB Schema | 254
PHP       | 7.2.6
MySQL     | 5.5.56-MariaDB
RRDTool   | 1.6.0
SNMP      | NET-SNMP 5.7.2
====================================

[OK]    Composer Version: 1.6.5
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[FAIL]  Discovery has not completed in the last 24 hours.
	[FIX] Check the cron job to make sure it is running and using discovery-wrapper.py
[WARN]  IPv6 is disabled on your server, you will not be able to add IPv6 devices.

[root@globlibrenms01 librenms]# ./validate.php -g distributedpoller
====================================
Component | Version
--------- | -------
LibreNMS  | 1.41-36-g67c585b
DB Schema | 254
PHP       | 7.2.6
MySQL     | 5.5.56-MariaDB
RRDTool   | 1.6.0
SNMP      | NET-SNMP 5.7.2
====================================

[OK]    Composer Version: 1.6.5
[OK]    Dependencies up-to-date.
Checking distributedpoller: OK
[OK]    Connection to memcached is ok

[root@globlibrenms01 librenms]# ./validate.php -g rrdcheck
====================================
Component | Version
--------- | -------
LibreNMS  | 1.41-36-g67c585b
DB Schema | 254
PHP       | 7.2.6
MySQL     | 5.5.56-MariaDB
RRDTool   | 1.6.0
SNMP      | NET-SNMP 5.7.2
====================================

[OK]    Composer Version: 1.6.5
[OK]    Dependencies up-to-date.
Checking rrdcheck:
Scanning 43354 rrd files in /opt/librenms/rrd...
Status: 43354/43354 - Complete
 OK

Any help is appreciated :slight_smile:

check in logs/librenms.log for any clues

also try turn distributed poller to false in your config.

Moving from distributed to local poller doesn’t seem to do anything - polling seems to happen either way, and RRD files are still being created / updated each poll.

The librenms log just has polling results, plus a some SQL errrors:
This is printed a bunch of times:

2018-07-18 20:50:41 MySQL Error: Cannot add or update a child row: a foreign key constraint fails (`librenms`.`component_statuslog`, CONSTRAINT `component_statuslog_ibfk_1` FOREIGN KEY (`component_id`) REFERENCES `component` (`id`) ON DELETE CASCADE ON UPDATE CASCADE) (INSERT INTO `component_statuslog` (`component_id`,`status`,`message`)  VALUES ('','1','Pool Member is Down: /Common/tcp_half_open: Connection reset @2018/07/18 19:51:37.'))

Then this is printed a bunch of times:

2018-07-18 20:50:41 MySQL Error: Cannot add or update a child row: a foreign key constraint fails (`librenms`.`component_statuslog`, CONSTRAINT `component_statuslog_ibfk_1` FOREIGN KEY (`component_id`) REFERENCES `component` (`id`) ON DELETE CASCADE ON UPDATE CASCADE) (INSERT INTO `component_statuslog` (`component_id`,`status`,`message`)  VALUES ('','2','The children pool member(s) are down'))

it looks like it happens every poll cycle.

I found the issue
I turned on HTTPS yesterday, and it turns I was loading an old tab that was missing the https: prefix, which was breaking images and cookies on the page.

For the amount of time I spent on troubleshooting this, I feel quite silly :sweat_smile:

1 Like