After much discussion on discord yesterday i decided to go to graylog to see if having graylog/nginx with sub directory was correctly configured.
Here is the discussion i had, a lot of goodies for testing 
davama - Yesterday at 9:41 AM
Hello, trying to integrate graylog into librenms but i have not success. 
not sure what do test now. Anyone able to have a peak? Graylog api access through nginx proxy - #3 by davama
LibreNMS Community
Graylog api access through nginx proxy
I’ve changed port 8888 to port 80 now but no change 
Even if i do not set the graylog port since http uses port 80 nothing happens.
Thank you!
laf - Yesterday at 9:47 AM
@davama Is base_uri correct, it’s changed numerous times in graylog
davama - Yesterday at 9:48 AM
@laf thank you for replying. Yes, it is. I know that because i can query the API via CML from the librenms box. i even set it manually
graylog config rest_listen_uri = http://127.0.0.1:9000/graylog/api/
but it’s behind nginx proxy…so idk if that has an issue with librenms
murrant - Yesterday at 9:50 AM
@davama no issues with proxies. We simply call the API.
davama - Yesterday at 9:50 AM
and this is nginx: proxy_pass http://127.0.0.1:9000/graylog;
laf - Yesterday at 9:50 AM
@davama open developer tools, go to network requests, refresh page and look for ajax_table.php
click that and see what output is shown
murrant - Yesterday at 9:51 AM
@davama are you double appending /graylog then?
zombah - Yesterday at 9:52 AM
@davama only proxy_pass? i have also proxy_set_header’s
davama - Yesterday at 9:52 AM
@murrant i thought that but no change regardless how i write it.
all the other header stuff is there too @zombah
murrant - Yesterday at 9:52 AM
@davama you need to see the actual url LibreNMS is trying and compare that to one that works.
davama - Yesterday at 9:53 AM
@murrant agreed!
this is from librenms trying to query.
nginx log:
2607:2400:901:36::189 - config [29/Jan/2018:14:44:36 +0000] “GET /graylog/api/?query=%2A&range=&limit=25&offset=0 HTTP/1.0” 200 232 “-” “-” “-”
curl
2607:2400:901:36::189 - config [29/Jan/2018:14:44:31 +0000] “GET /graylog/api/cluster HTTP/1.1” 200 420 “-” “curl/7.29.0” “-”
curl outputs the cluster id, codename,etc
@laf i dont know where that page is
laf - Yesterday at 9:55 AM
@davama I’m sure you can google how to use developer tools 
davama - Yesterday at 9:55 AM
@laf got it! thought is was some page in libre. ok let me try that
thank you for the input!
davama - Yesterday at 10:01 AM
hmmm
not sure is this is what i need to look at but this is the Request Headers
:authority:librenms-lab
:method:POST
:path:/ajax_table.php
:scheme:https
accept:/
accept-encoding:gzip, deflate, br
accept-language:en-US,en;q=0.9
content-length:63
content-type:application/x-www-form-urlencoded; charset=UTF-8
cookie:PHPSESSID=t9e8u250j1977dca6g0mlong04
origin:https://librenms-lab
referer:https://librenms-lab/graylog/
user-agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36
x-requested-with:XMLHttpRequest
laf - Yesterday at 10:03 AM
You should have an output or response tab, look at that
davama - Yesterday at 10:03 AM
Thank you @laf
{
“current”: 1,
“rowCount”: 25,
“rows”: ,
“total”: 0
}
laf - Yesterday at 10:05 AM
So no data, you sure you’re searching for something that exists?
davama - Yesterday at 10:05 AM
My filter is all devices, so i dont know if that matters
and yes, my devices have logged in the last 5 minutes
graylog and librenms are two seperate boxes. if that was not clear
graylog is accessible via web with https but allowed via specific host-ip via http
when i try the same query that i see librenms do with curl i get:
{
“cluster_id”: “2b80bbb0-23db-42e6-9f7d-61a631beaec1”,
“node_id”: “fecd1603-dc68-4ac1-a19e-f425cd749b3f”,
“tagline”: “Manage your logs in the dark and have lasers going and make it look like you’re from space!”,
“version”: “2.4.3+2c41897”
}
laf - Yesterday at 10:10 AM
I think you’re testing against a different api call than we use
murrant - Yesterday at 10:11 AM
@davama btw, my graylog nginx uses this config (much removed):
location / {
proxy_pass http://127.0.0.1:9000;
}
davama - Yesterday at 10:11 AM
@laf i just tryied what i saw the log say in nginx
@murrant
mine is
location /graylog {
blabla
proxy_pass http://127.0.0.1:9000/graylog
laf - Yesterday at 10:12 AM
@davama run php -a
paste in your config.php graylog config
then this:
if (isset($config[‘graylog’][‘base_uri’])) {
$graylog_base = $config[‘graylog’][‘base_uri’];
} elseif (version_compare($config[‘graylog’][‘version’], ‘2.1’, ‘>=’)) {
$graylog_base = ‘/api/search/universal/relative’;
} else {
$graylog_base = ‘/search/universal/relative’;
}
$graylog_url = $config[‘graylog’][‘server’] . ‘:’ . $config[‘graylog’][‘port’] . $graylog_base . ‘?query=’ . urlencode($query) . ‘&range=’. $filter_range . $extra_query;
print_r($graylog_url);
And see what’s printed out
davama - Yesterday at 10:14 AM
@laf ok one sec.
thank you all again for the input! this has caused me a lot of headaches
@laf the print gave me this
http://usdsb.nwk.jwm2.net:80/graylog/api/?query=&range=
which i tried with curl and output was the same space stuff
murrant - Yesterday at 10:16 AM
try that with curl
davama - Yesterday at 10:16 AM
@murrant same space stuff output
http://usdsb.nwk.jwm2.net/graylog/api/cluster this works fine… 
same as this http://usdsb.nwk.jwm2.net/graylog/api/cluster
http://usdsb.nwk.jwm2.net:80/graylog/api/cluster
im stumped
murrant - Yesterday at 10:18 AM
try this url in curl http://usdsb.nwk.jwm2.net:80/graylog/api/?query=*
davama - Yesterday at 10:20 AM
@murrant same space output
laf - Yesterday at 10:24 AM
What does same space output mean?
zombah - Yesterday at 10:26 AM
probably graylog tagline about lasers from space
davama - Yesterday at 10:26 AM
yes, sorry that output tagline
laf - Yesterday at 10:27 AM
I don’t do graylog so I don’t know what tagline is
murrant - Yesterday at 10:27 AM
@davama This is what it should look like /api/search/universal/relative?query=%2A
@laf it is some random text that means we didn’t find anything.
davama - Yesterday at 10:28 AM
@murrant
http://usdsb.nwk.jwm2.net:80/graylog/api/search/universal/relative?query=*
{
“message”: “Not authorized”,
“type”: “ApiError”
}
laf - Yesterday at 10:28 AM
The docs look wrong for graylog
@davama remove the base_uri config
murrant - Yesterday at 10:29 AM
@davama unset $config[‘graylog’][‘base_uri’] in your config.php
laf - Yesterday at 10:29 AM
You’ve specified 2.4.0 so the base uri should be what murrant posted and will be auto detected
davama - Yesterday at 10:30 AM
@laf i’ve since upgraded to 2.4.3 which i reflect in my config.php
let me try the base_uri thing
laf - Yesterday at 10:30 AM
That’s fine still unless graylog have again changed the api url which I doubt
davama - Yesterday at 10:30 AM
@laf i assume i have to change the server to blah/graylog no?
murrant - Yesterday at 10:31 AM
no
laf - Yesterday at 10:31 AM
@davama no
just set it to to http://hostname:portnumber(edited)
drop :portnumber if it’s 80
or 443 if https
murrant - Yesterday at 10:32 AM
@davama once you drop base_uri, now your LibreNMS config is correct and your nginx config is incorrect.
graylog changed the api to be under /api and your nginx config doesn’t pass that.
davama - Yesterday at 10:33 AM
$config[‘graylog’][‘server’] = ‘http://usdsb.nwk.jwm2.net:80’;
#removed base_uri and port
same Responce from the ajax_table.php
@murrant let me change nginx
ok ok
when i remove port from the config.php i dont see the graylog option in overview
need to have the port set.
so then this means i have the port 80 configured twice
murrant - Yesterday at 10:35 AM
@davama you need to have server and port split out, laf was speaking from memory
davama - Yesterday at 10:37 AM
@murrant thank you for clarifying
nginx config:
location /graylog {
access_log /var/log/nginx/graylog-access.log main;
error_log /var/log/nginx/graylog-error.log debug;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Graylog-Server-URL http://$server_name/graylog/api;
proxy_pass http://127.0.0.1:9000/graylog;
}
murrant - Yesterday at 10:39 AM
If you want to keep that config you can set the base_uri in LibreNMS to /graylog/api/search/universal/relative that might work.
otherwise, drop all the graylog sub directories.
davama - Yesterday at 10:41 AM
@murrant tried you suggested base_uri but no change
let me try dropping the sub directories
thank again for the kind help and patience
here is my new nginx… minor change. it’s accessible via web browser
location / {
access_log /var/log/nginx/graylog-access.log main;
error_log /var/log/nginx/graylog-error.log debug;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Graylog-Server-URL http://$server_name/graylog/api;
#proxy_pass http://127.0.0.1:9000/graylog;
proxy_pass http://127.0.0.1:9000;
}
librenms config.php
$config[‘graylog’][‘server’] = ‘http://usdsb.nwk.jwm2.net’;
// $config[‘graylog’][‘base_uri’] = ‘/graylog/api/’;
// $config[‘graylog’][‘base_uri’] = ‘/graylog/api/search/universal/relative’;
$config[‘graylog’][‘port’] = ‘80’;
$config[‘graylog’][‘username’] = ‘config’;
$config[‘graylog’][‘password’] = ‘password’;
$config[‘graylog’][‘version’] = ‘2.4.3’;
$config[‘graylog’][‘timezone’] = ‘UTC’;
nginx log:
2607:2400:901:36::189 - config [29/Jan/2018:15:52:34 +0000] “GET /api/search/universal/relative?query=%2A&range=&limit=25&offset=0 HTTP/1.0” 404 50 “-” “-” “-”
unfortunately no change
murrant - Yesterday at 10:55 AM
@davama if you try that url in curl?
davama - Yesterday at 10:56 AM
{
“message”: “HTTP 404 Not Found”,
“type”: “ApiError”
}
http://usdsb.nwk.jwm2.net/graylog/api/search/universal/relative?query=*&range=&limit=25&offset=0
http://usdsb.nwk.jwm2.net:80/graylog/api/search/universal/relative?query=*&range=&limit=25&offset=0
http://usdsb.nwk.jwm2.net:80/search/universal/relative?query=*&range=&limit=25&offset=0
http://usdsb.nwk.jwm2.net:80/api/search/universal/relative?query=*&range=&limit=25&offset=0
try all
tried*
murrant - Yesterday at 10:57 AM
what is rest_listen_uri and rest_transport_uri in your graylog server.conf?
davama - Yesterday at 10:58 AM
rest_listen_uri = http://127.0.0.1:9000/graylog/api/
#rest_transport_uri = http://192.168.1.1:9000/api/
web_listen_uri = http://127.0.0.1:9000/graylog
we host several web services on the graylog server so that is why we have sub directories in nginx
but all hosted via https
murrant - Yesterday at 11:00 AM
Here is mine:
rest_listen_uri = http://127.0.0.1:9000/api/
rest_transport_uri = https://logs.mydomain.com/api/
subdomains > subdirectories but anyway
set this:
rest_listen_uri = http://127.0.0.1:9000/api/
rest_transport_uri = https://usdsb.nwk.jwm2.net/graylog/api/
location /graylog {
proxy_set_header X-Graylog-Server-URL https://$server_name/graylog/api;
proxy_pass http://127.0.0.1:9000;
}
(edited)
$config[‘graylog’][‘base_uri’] = ‘/graylog/api/search/universal/relative’;
davama - Yesterday at 11:08 AM
trying…
what about the web_uri?
murrant - Yesterday at 11:10 AM
probably needs to be https://usdsb.nwk.jwm2.net/graylog right?
btw, I updated X-Graylog-Server-URL to be https above
sorry, it should be web_listen_uri = http://127.0.0.1:9000 Didn’t know you meant the listen uri(edited)
davama - Yesterday at 11:14 AM
ah
thanks!
graylog changes:
rest_listen_uri = http://127.0.0.1:9000/api/
rest_transport_uri = https://usdsb.nwk.jwm2.net/graylog/api/
web_listen_uri = http://127.0.0.1:9000
nginx
proxy_set_header X-Graylog-Server-URL https://$server_name/graylog/api;
proxy_pass http://127.0.0.1:9000;
which makes me lose access via web using the subdirectory
still playing around
murrant - Yesterday at 11:22 AM
you probably need a rewrite rule in nginx
try: rewrite ^/graylog(.*)$ $1 last;
davama - Yesterday at 11:32 AM
thank you @murrant for the suggestions
when i use / in nginx im able to access it… which is fine…
web access to both of these work finr:
https://usdsb…/
http://usdsb…/
but when i try librenms… no dice still
murrant - Yesterday at 11:33 AM
well, you need graylog / nginx to work right first
did you try the rewrite rule? should allow you to access via subdir
davama - Yesterday at 11:34 AM
i did but i could not access via subdirectory via web
i have two graylog.conf nginx files
one with subdir (port 80)
and one with / (port 443)
murrant - Yesterday at 11:35 AM
?
davama - Yesterday at 11:35 AM
each referenced in it’s respective server block in nginx
the one we use to access graylog via web browser is 443 https
we trying to use port 80 to access for internal things (like librenms)
both behind nginx proxy
hope that makes sense
murrant - Yesterday at 11:37 AM
@davama here is y complete nginx config, for comparison Untitled - LibreNMS
no reason to use port 80 internally in my opnion
davama - Yesterday at 11:39 AM
that’s true… but since i’ve tried as much as i can think of…wanted to remove the https out of the equation…really would like to make this work but need with subdir
ill look at your config…
thank you !!!
davama - Yesterday at 11:59 AM
@laf @murrant i think it would be better to find out first, how to better configure graylog nginx with subdirectories, then tackle this librenms issues…i feel that’s probably the main issue here
davama - Yesterday at 12:00 PM
thank you both for the kind help
ill post what i’ve done for my sake
murrant - Yesterday at 12:00 PM
@davama indeed, sorry, I tried to guess for you. Post it on the community site. for now @davama
davama - Yesterday at 12:01 PM
That’s what ill do.
thank you very much!strong text
So took it to graylog. The discussion is here: Graylog behind nginx sub directory how-to configure - #8 by jochen - Graylog Central (peer support) - Graylog Community
Looks like that is fine too…
Next test is to build from scratch without sub directory but still behind nginx…
