Graylog api access through nginx proxy

Hello

Im trying to integrate graylog into librenms but no beans :frowning:

Current librenms enviroment

Version 1.35-97-g788c0bd - Sun Jan 14 2018 15:43:55 GMT-0500
DB Schema #229
Web Server nginx/1.12.2
PHP 7.0.25
MySQL 5.5.56-MariaDB
RRDtool 1.6.0

Graylog version 2.4.0+2115a42 (note: separate host from librenms)

Both are Centos 7.4.1708 with latest kernel 3.10.0-693.11.6.el7.x86_64

Current /opt/librenms/config.php graylog settings:

$config['graylog']['server'] = 'http://<fqdn>';
$config['graylog']['base_uri'] = '/graylog/api/';
$config['graylog']['port'] = '8888';
$config['graylog']['username'] = 'config';
$config['graylog']['password'] = '<password>';
$config['graylog']['version'] = '2.4.0';
$config['graylog']['timezone'] = 'UTC';

on version i’ve also tried 2.4.0+2115a42 but nothing

When i go to Overview>Graylog i get an empty page
On my nginx access.log i see

<librenms-ipv6-address> - config [19/Jan/2018:13:50:17 +0000] "GET /graylog/api/?query=%2A&range=&limit=25&offset=0 HTTP/1.0" 200 232 "-" "-" "-"

Now if i curl from librenms host CML

curl -X GET -u "config:<password>" -H 'Accept: application/json' 'http://<fqdn>:8888/graylog/api/?query=%2A&range=&limit=25&offset=0' | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   232  100   232    0     0  26393      0 --:--:-- --:--:-- --:--:-- 29000
{
    "cluster_id": "2b80bbb0-23db-42e6-9f7d-61a631beaec1",
    "node_id": "fecd1603-dc68-4ac1-a19e-f425cd749b3f",
    "tagline": "Manage your logs in the dark and have lasers going and make it look like you're from space!",
    "version": "2.4.0+2115a42"
}

That outputs a similar log in nginx

<librenms-ipv6-address> - config [19/Jan/2018:14:07:20 +0000] "GET /graylog/api/?query=%2A&range=&limit=25&offset=0 HTTP/1.1" 200 232 "-" "curl/7.29.0" "-"

My validate.php output is almost perfect except

Warning: Your install is over 24 hours out of date, last update: Sun, 14 Jan 2018 20:43:55 +0000
Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.

But dont think that is relevant.

I am at a lost as to what else to try :cry:
Any input is appreciated.

thanks you,
dave

Update:

Debugging nginx…
Curl command from librenms host

$ curl -4 -X GET -u "config:<password>" -H 'Accept: application/json' 'http://<graylog-host>:8888/graylog/api/cluster' | python -m json.tool
{
    "fecd1603-dc68-4ac1-a19e-f425cd749b3f": {
        "cluster_id": "2b80bbb0-23db-42e6-9f7d-61a631beaec1",
        "codename": "Wildwuchs",
        "facility": "graylog-server",
        "hostname": "<redacted-ip>",
        "is_processing": true,
        "lb_status": "alive",
        "lifecycle": "running",
        "node_id": "fecd1603-dc68-4ac1-a19e-f425cd749b3f",
        "operating_system": "Linux 3.10.0-693.11.6.el7.x86_64",
        "started_at": "2018-01-18T14:19:41.945Z",
        "timezone": "UTC",
        "version": "2.4.0+2115a42"
    }
}

Using the same graylog settings as posted before.

I did a vimdiff on all three ways graylog api is queried (librenms web gui, curl-equivalent, curl-cluster) but i dont see anything interesting

I’ve changed port 8888 to port 80 now but no change :frowning:

Even if i do not set the graylog port since http uses port 80 nothing happens.

After much discussion on discord yesterday i decided to go to graylog to see if having graylog/nginx with sub directory was correctly configured.

Here is the discussion i had, a lot of goodies for testing :slight_smile:

davama - Yesterday at 9:41 AM
Hello, trying to integrate graylog into librenms but i have not success. :frowning: not sure what do test now. Anyone able to have a peak? Graylog api access through nginx proxy - #3 by davama
LibreNMS Community
Graylog api access through nginx proxy
I’ve changed port 8888 to port 80 now but no change :frowning_with_open_mouth: Even if i do not set the graylog port since http uses port 80 nothing happens.

Thank you!

laf - Yesterday at 9:47 AM
@davama Is base_uri correct, it’s changed numerous times in graylog

davama - Yesterday at 9:48 AM
@laf thank you for replying. Yes, it is. I know that because i can query the API via CML from the librenms box. i even set it manually
graylog config rest_listen_uri = http://127.0.0.1:9000/graylog/api/
but it’s behind nginx proxy…so idk if that has an issue with librenms

murrant - Yesterday at 9:50 AM
@davama no issues with proxies. We simply call the API.

davama - Yesterday at 9:50 AM
and this is nginx: proxy_pass http://127.0.0.1:9000/graylog;

laf - Yesterday at 9:50 AM
@davama open developer tools, go to network requests, refresh page and look for ajax_table.php
click that and see what output is shown

murrant - Yesterday at 9:51 AM
@davama are you double appending /graylog then?

zombah - Yesterday at 9:52 AM
@davama only proxy_pass? i have also proxy_set_header’s

davama - Yesterday at 9:52 AM
@murrant i thought that but no change regardless how i write it.
all the other header stuff is there too @zombah

murrant - Yesterday at 9:52 AM
@davama you need to see the actual url LibreNMS is trying and compare that to one that works.

davama - Yesterday at 9:53 AM
@murrant agreed!
this is from librenms trying to query.
nginx log:
2607:2400:901:36::189 - config [29/Jan/2018:14:44:36 +0000] “GET /graylog/api/?query=%2A&range=&limit=25&offset=0 HTTP/1.0” 200 232 “-” “-” “-”
curl
2607:2400:901:36::189 - config [29/Jan/2018:14:44:31 +0000] “GET /graylog/api/cluster HTTP/1.1” 200 420 “-” “curl/7.29.0” “-”
curl outputs the cluster id, codename,etc
@laf i dont know where that page is

laf - Yesterday at 9:55 AM
@davama I’m sure you can google how to use developer tools :smiley:

davama - Yesterday at 9:55 AM
@laf got it! thought is was some page in libre. ok let me try that
thank you for the input!

davama - Yesterday at 10:01 AM
hmmm
not sure is this is what i need to look at but this is the Request Headers
:authority:librenms-lab
:method:POST
:path:/ajax_table.php
:scheme:https
accept:/
accept-encoding:gzip, deflate, br
accept-language:en-US,en;q=0.9
content-length:63
content-type:application/x-www-form-urlencoded; charset=UTF-8
cookie:PHPSESSID=t9e8u250j1977dca6g0mlong04
origin:https://librenms-lab
referer:https://librenms-lab/graylog/
user-agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36
x-requested-with:XMLHttpRequest

laf - Yesterday at 10:03 AM
You should have an output or response tab, look at that

davama - Yesterday at 10:03 AM
Thank you @laf
{
“current”: 1,
“rowCount”: 25,
“rows”: ,
“total”: 0
}

laf - Yesterday at 10:05 AM
So no data, you sure you’re searching for something that exists?

davama - Yesterday at 10:05 AM
My filter is all devices, so i dont know if that matters
and yes, my devices have logged in the last 5 minutes
graylog and librenms are two seperate boxes. if that was not clear
graylog is accessible via web with https but allowed via specific host-ip via http
when i try the same query that i see librenms do with curl i get:
{
“cluster_id”: “2b80bbb0-23db-42e6-9f7d-61a631beaec1”,
“node_id”: “fecd1603-dc68-4ac1-a19e-f425cd749b3f”,
“tagline”: “Manage your logs in the dark and have lasers going and make it look like you’re from space!”,
“version”: “2.4.3+2c41897”
}

laf - Yesterday at 10:10 AM
I think you’re testing against a different api call than we use

murrant - Yesterday at 10:11 AM
@davama btw, my graylog nginx uses this config (much removed):
location / {
proxy_pass http://127.0.0.1:9000;
}

davama - Yesterday at 10:11 AM
@laf i just tryied what i saw the log say in nginx
@murrant
mine is
location /graylog {
blabla
proxy_pass http://127.0.0.1:9000/graylog

laf - Yesterday at 10:12 AM
@davama run php -a
paste in your config.php graylog config
then this:
if (isset($config[‘graylog’][‘base_uri’])) {
$graylog_base = $config[‘graylog’][‘base_uri’];
} elseif (version_compare($config[‘graylog’][‘version’], ‘2.1’, ‘>=’)) {
$graylog_base = ‘/api/search/universal/relative’;
} else {
$graylog_base = ‘/search/universal/relative’;
}

$graylog_url = $config[‘graylog’][‘server’] . ‘:’ . $config[‘graylog’][‘port’] . $graylog_base . ‘?query=’ . urlencode($query) . ‘&range=’. $filter_range . $extra_query;
print_r($graylog_url);
And see what’s printed out

davama - Yesterday at 10:14 AM
@laf ok one sec.

thank you all again for the input! this has caused me a lot of headaches
@laf the print gave me this
http://usdsb.nwk.jwm2.net:80/graylog/api/?query=&range=
which i tried with curl and output was the same space stuff

murrant - Yesterday at 10:16 AM
try that with curl

davama - Yesterday at 10:16 AM
@murrant same space stuff output
http://usdsb.nwk.jwm2.net/graylog/api/cluster this works fine… :frowning2:
same as this http://usdsb.nwk.jwm2.net/graylog/api/cluster
http://usdsb.nwk.jwm2.net:80/graylog/api/cluster
im stumped :frowning:

murrant - Yesterday at 10:18 AM
try this url in curl http://usdsb.nwk.jwm2.net:80/graylog/api/?query=*

davama - Yesterday at 10:20 AM
@murrant same space output

laf - Yesterday at 10:24 AM
What does same space output mean?

zombah - Yesterday at 10:26 AM
probably graylog tagline about lasers from space

davama - Yesterday at 10:26 AM
yes, sorry that output tagline

laf - Yesterday at 10:27 AM
I don’t do graylog so I don’t know what tagline is

murrant - Yesterday at 10:27 AM
@davama This is what it should look like /api/search/universal/relative?query=%2A
@laf it is some random text that means we didn’t find anything.

davama - Yesterday at 10:28 AM
@murrant
http://usdsb.nwk.jwm2.net:80/graylog/api/search/universal/relative?query=*
{
“message”: “Not authorized”,
“type”: “ApiError”
}

laf - Yesterday at 10:28 AM
The docs look wrong for graylog
@davama remove the base_uri config

murrant - Yesterday at 10:29 AM
@davama unset $config[‘graylog’][‘base_uri’] in your config.php

laf - Yesterday at 10:29 AM
You’ve specified 2.4.0 so the base uri should be what murrant posted and will be auto detected
davama - Yesterday at 10:30 AM
@laf i’ve since upgraded to 2.4.3 which i reflect in my config.php
let me try the base_uri thing

laf - Yesterday at 10:30 AM
That’s fine still unless graylog have again changed the api url which I doubt

davama - Yesterday at 10:30 AM
@laf i assume i have to change the server to blah/graylog no?

murrant - Yesterday at 10:31 AM
no

laf - Yesterday at 10:31 AM
@davama no
just set it to to http://hostname:portnumber(edited)
drop :portnumber if it’s 80
or 443 if https

murrant - Yesterday at 10:32 AM
@davama once you drop base_uri, now your LibreNMS config is correct and your nginx config is incorrect.
graylog changed the api to be under /api and your nginx config doesn’t pass that.

davama - Yesterday at 10:33 AM
$config[‘graylog’][‘server’] = ‘http://usdsb.nwk.jwm2.net:80’;
#removed base_uri and port
same Responce from the ajax_table.php
@murrant let me change nginx
ok ok
when i remove port from the config.php i dont see the graylog option in overview
need to have the port set.
so then this means i have the port 80 configured twice

murrant - Yesterday at 10:35 AM
@davama you need to have server and port split out, laf was speaking from memory

davama - Yesterday at 10:37 AM
@murrant thank you for clarifying
nginx config:
location /graylog {
access_log /var/log/nginx/graylog-access.log main;
error_log /var/log/nginx/graylog-error.log debug;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Graylog-Server-URL http://$server_name/graylog/api;
proxy_pass http://127.0.0.1:9000/graylog;
}

murrant - Yesterday at 10:39 AM
If you want to keep that config you can set the base_uri in LibreNMS to /graylog/api/search/universal/relative that might work.
otherwise, drop all the graylog sub directories.

davama - Yesterday at 10:41 AM
@murrant tried you suggested base_uri but no change :frowning:

let me try dropping the sub directories
thank again for the kind help and patience

here is my new nginx… minor change. it’s accessible via web browser
location / {
access_log /var/log/nginx/graylog-access.log main;
error_log /var/log/nginx/graylog-error.log debug;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Graylog-Server-URL http://$server_name/graylog/api;
#proxy_pass http://127.0.0.1:9000/graylog;
proxy_pass http://127.0.0.1:9000;
}

librenms config.php
$config[‘graylog’][‘server’] = ‘http://usdsb.nwk.jwm2.net’;
// $config[‘graylog’][‘base_uri’] = ‘/graylog/api/’;
// $config[‘graylog’][‘base_uri’] = ‘/graylog/api/search/universal/relative’;
$config[‘graylog’][‘port’] = ‘80’;
$config[‘graylog’][‘username’] = ‘config’;
$config[‘graylog’][‘password’] = ‘password’;
$config[‘graylog’][‘version’] = ‘2.4.3’;
$config[‘graylog’][‘timezone’] = ‘UTC’;

nginx log:
2607:2400:901:36::189 - config [29/Jan/2018:15:52:34 +0000] “GET /api/search/universal/relative?query=%2A&range=&limit=25&offset=0 HTTP/1.0” 404 50 “-” “-” “-”

unfortunately no change

murrant - Yesterday at 10:55 AM
@davama if you try that url in curl?

davama - Yesterday at 10:56 AM
{
“message”: “HTTP 404 Not Found”,
“type”: “ApiError”
}
http://usdsb.nwk.jwm2.net/graylog/api/search/universal/relative?query=*&range=&limit=25&offset=0
http://usdsb.nwk.jwm2.net:80/graylog/api/search/universal/relative?query=*&range=&limit=25&offset=0
http://usdsb.nwk.jwm2.net:80/search/universal/relative?query=*&range=&limit=25&offset=0
http://usdsb.nwk.jwm2.net:80/api/search/universal/relative?query=*&range=&limit=25&offset=0
try all
tried*

murrant - Yesterday at 10:57 AM
what is rest_listen_uri and rest_transport_uri in your graylog server.conf?

davama - Yesterday at 10:58 AM
rest_listen_uri = http://127.0.0.1:9000/graylog/api/
#rest_transport_uri = http://192.168.1.1:9000/api/
web_listen_uri = http://127.0.0.1:9000/graylog
we host several web services on the graylog server so that is why we have sub directories in nginx
but all hosted via https

murrant - Yesterday at 11:00 AM
Here is mine:
rest_listen_uri = http://127.0.0.1:9000/api/
rest_transport_uri = https://logs.mydomain.com/api/
subdomains > subdirectories :smiley: but anyway
set this:
rest_listen_uri = http://127.0.0.1:9000/api/
rest_transport_uri = https://usdsb.nwk.jwm2.net/graylog/api/
location /graylog {
proxy_set_header X-Graylog-Server-URL https://$server_name/graylog/api;
proxy_pass http://127.0.0.1:9000;
}
(edited)
$config[‘graylog’][‘base_uri’] = ‘/graylog/api/search/universal/relative’;

davama - Yesterday at 11:08 AM
trying…
what about the web_uri?

murrant - Yesterday at 11:10 AM
probably needs to be https://usdsb.nwk.jwm2.net/graylog right?
btw, I updated X-Graylog-Server-URL to be https above
sorry, it should be web_listen_uri = http://127.0.0.1:9000 Didn’t know you meant the listen uri(edited)

davama - Yesterday at 11:14 AM
ah
thanks!
graylog changes:

rest_listen_uri = http://127.0.0.1:9000/api/
rest_transport_uri = https://usdsb.nwk.jwm2.net/graylog/api/
web_listen_uri = http://127.0.0.1:9000

nginx
proxy_set_header X-Graylog-Server-URL https://$server_name/graylog/api;
proxy_pass http://127.0.0.1:9000;

which makes me lose access via web :frowning: using the subdirectory
still playing around

murrant - Yesterday at 11:22 AM
you probably need a rewrite rule in nginx
try: rewrite ^/graylog(.*)$ $1 last;

davama - Yesterday at 11:32 AM
thank you @murrant for the suggestions

when i use / in nginx im able to access it… which is fine…
web access to both of these work finr:
https://usdsb…/
http://usdsb…/
but when i try librenms… no dice still

murrant - Yesterday at 11:33 AM
well, you need graylog / nginx to work right first :smiley:
did you try the rewrite rule? should allow you to access via subdir

davama - Yesterday at 11:34 AM
i did but i could not access via subdirectory via web
i have two graylog.conf nginx files
one with subdir (port 80)
and one with / (port 443)

murrant - Yesterday at 11:35 AM
?

davama - Yesterday at 11:35 AM
each referenced in it’s respective server block in nginx
the one we use to access graylog via web browser is 443 https
we trying to use port 80 to access for internal things (like librenms)
both behind nginx proxy
hope that makes sense

murrant - Yesterday at 11:37 AM
@davama here is y complete nginx config, for comparison Untitled - LibreNMS
no reason to use port 80 internally in my opnion

davama - Yesterday at 11:39 AM
that’s true… but since i’ve tried as much as i can think of…wanted to remove the https out of the equation…really would like to make this work but need with subdir
ill look at your config…
thank you !!!

davama - Yesterday at 11:59 AM
@laf @murrant i think it would be better to find out first, how to better configure graylog nginx with subdirectories, then tackle this librenms issues…i feel that’s probably the main issue here

davama - Yesterday at 12:00 PM
thank you both for the kind help
ill post what i’ve done for my sake

murrant - Yesterday at 12:00 PM
@davama indeed, sorry, I tried to guess for you. Post it on the community site. for now @davama :smiley:

davama - Yesterday at 12:01 PM
That’s what ill do.
thank you very much!strong text

So took it to graylog. The discussion is here: Graylog behind nginx sub directory how-to configure - #8 by jochen - Graylog Central (peer support) - Graylog Community
Looks like that is fine too… :frowning:

Next test is to build from scratch without sub directory but still behind nginx…

Ok, ive since built a new graylog vm with nginx proxy.

Slight diff is that it’s in / and it’s not using SSL. ( trying to remove as many variables as possible)

See post: https://community.graylog.org/t/graylog-behind-nginx-sub-directory-how-to-configure/3968/10?u=davama

So i believe it’s an librenms issue. Graylog folks said it was fine with sub directories but now, at least with the VM setup, i am NOT using sub directories but root /

@jochen had mentioned about HTTP1.0 vs HTTP1.1 which was interesting because librenms DOES use HTTP1.0 … not sure if that has an issue but at least with curl it did not

Any input is always appreciated.
Thank you for the continued support!

Again my graylog enviroment is:

graylog 2.4.3+2c41897
nginx nginx/1.12.2
elasticsearch 5.6.7
mongodb v3.2.18

Librenms enviroment is:

Version 1.35-97-g788c0bd - Sun Jan 14 2018 15:43:55 GMT-0500
DB Schema #229
Web Server nginx/1.12.2
PHP 7.0.25
MySQL 5.5.56-MariaDB
RRDtool 1.6.0

config.php

// $config['graylog']['server'] = 'http://usdsb.nwk.jwm2.net';
$config['graylog']['server'] = 'http://graylog-lab-ecr-1';
//$config['graylog']['base_uri'] = '/graylog/api/';
$config['graylog']['base_uri'] = '/api/';
// $config['graylog']['base_uri'] = '/graylog/api/search/universal/relative';
$config['graylog']['port'] = '80';
$config['graylog']['username'] = 'config';
$config['graylog']['password'] = 'password';
$config['graylog']['version'] = '2.4.3';
$config['graylog']['timezone'] = 'UTC';

Which logs:

<ipv6> - config [30/Jan/2018:19:45:21 +0000] "GET /api/?query=%2A&range=&limit=25&offset=0 HTTP/1.0" 200 232 "-" "-" "-"

thanks!!

I really don’t think this is a librenms issue other than you’ve not got the correct config for your setup.

Many others use it with success.

OMG!!! I solved it!!!

Ok, so journey as to what i was doing…

So i decided to poke around the graylog api web interface inside the
GET /search/universal/relative section
URL: http://graylog-lab-ecr-1/api/api-browser#!/Search/Relative/searchRelativeChunked_get_0

Filled in the fields:

query source:h-txa-1ter-ecr-1
range 3600
fields source,message

Which gave me the Request URL:
http://graylog-lab-ecr-1:80/api/search/universal/relative?query=source%3Ah-txa-1ter-rtr-1&range=3600&fields=source%2Cmessage

with a Response Body:

"timestamp","source","message"
"2018-01-30T20:00:32.785Z","H-TXA-1TER-RTR-1","Jan 30 20:15:41.301 UTC: %SSH-5-SSH2_USERAUTH: User 'config' authentication for SSH2 Session from <ipv6> (tty = 0) using crypto cipher 'aes128-cbc', hmac 'hmac-sha1' Succeeded"
"2018-01-30T20:00:32.785Z","H-TXA-1TER-RTR-1","Jan 30 20:15:41.301 UTC: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: config] [Source: <ipv6>] [localport: 22] at 20:15:41 UTC Tue Jan 30 2018"
"2018-01-30T20:00:32.785Z","H-TXA-1TER-RTR-1","Jan 30 20:15:41.093 UTC: %SSH-5-SSH2_SESSION: SSH2 Session request from <ipv6> (tty = 0) using crypto cipher 'aes128-cbc', hmac 'hmac-sha1' Succeeded"
"2018-01-30T20:00:36.785Z","H-TXA-1TER-RTR-1","Jan 30 20:15:46.721 UTC: %SYS-6-LOGOUT: User config has exited tty session 2(<ipv6>)"
"2018-01-30T20:00:38.785Z","H-TXA-1TER-RTR-1","Jan 30 20:15:46.721 UTC: %SSH-5-SSH2_CLOSE: SSH2 Session from <ipv6> (tty = 0) for user 'config' using crypto cipher 'aes128-cbc', hmac 'hmac-sha1' closed"

So i fiddled a bit with that under curl. Which after much playing around led me to this output:

{
    "message": "Not authorized",
    "type": "ApiError"
}

Which led me to this post in google: https://community.graylog.org/t/rest-api-search-authorization/1103

Which led me to use the credentials of the graylog administrator.

curl -4 -X GET -u "administrator" -H 'Accept: application/json' 'http://graylog-lab-ecr-1/api/search/universal/relative?query=%2A&range=&limit=25&offset=0' | python -m json.tool
Enter host password for user 'administrator':
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    75  100    75    0     0   2675      0 --:--:-- --:--:-- --:--:--  2777
{
    "message": "Invalid credentials in Authorization header",
    "type": "ApiError"
}

This led me to believe that it was the colon inside my admin password that did not allow me to access the api.
Change it with a non-colon and … BOOM!!!

curl -4 -X GET -u "administrator:administrator" -H 'Accept: application/json' 'http://graylog-lab-ecr-1/api/search/universal/relative?query=%2A&range=&limit=25&offset=0' | python -m json.tool
 "decoration_stats": null,
    "fields": [
        "application_name",
        "level",
        "source",
        "ip_address",
        "message",
        "facility",
        "timestamp"
    ],
    "from": "1970-01-01T00:00:00.000Z",
    "messages": [
        {
            "decoration_stats": null,
            "highlight_ranges": {},
            "index": "graylog2_0",
            "message": {
                "_id": "3096b240-05fc-11e8-9ffd-525400418107",
                "application_name": "systemd",
                "facility": "daemon",
                "gl2_remote_ip": "127.0.0.1",
                "gl2_remote_port": 47594,
                "gl2_source_input": "58a4b2b948fdda09423bcced",
                "gl2_source_node": "bc808e5c-1271-4d83-b3ff-160d52e37433",
                "ip_address": "127.0.0.1",
                "level": "6",
                "message": "Stopping Graylog server...",
                "source": "graylog-lab-ecr-1",
                "streams": [
                    "58803ca348fdda55811e9ca0",
                    "000000000000000000000001"
                ],
                "timestamp": "2018-01-30T20:28:55.073Z"
            }
        },
        {
            "decoration_stats": null,
            "highlight_ranges": {},
            "index": "graylog2_0",
            "message": {
                "_id": "3096d950-05fc-11e8-9ffd-525400418107",
                "application_name": "polkitd",
                "facility": "authpriv",
                "gl2_remote_ip": "127.0.0.1",
                "gl2_remote_port": 47594,
                "gl2_source_input": "58a4b2b948fdda09423bcced",
                "gl2_source_node": "bc808e5c-1271-4d83-b3ff-160d52e37433",
                "ip_address": "127.0.0.1",
                "level": "5",
                "message": "Registered Authentication Agent for unix-process:3411:773421 (system bus name :1.33 [\\/usr\\/bin\\/pkttyagent --notify-fd 5 --fallback], object path \\/org\\/freedesktop\\/PolicyKit1\\/AuthenticationAgent, locale en_US.UTF-8)",
                "source": "graylog-lab-ecr-1",
                "streams": [
                    "58803ca348fdda55811e9ca0",
                    "000000000000000000000001"
                ],
                "timestamp": "2018-01-30T20:28:55.072Z"
...
...
...

SOO lesson learned???

  • To access the API universal search you need to use admin (couldn’t find in the graylog doc where it said that though) (would probably explore the non-admin approach)…but the librenms doc had it … :man_facepalming:
  • DO NOT use a colon : on your password…

What a headache! But fun!
Glad it’s working!!! Woohoo!!! :sunglasses: