Microsoft Teams Transport - Failed (General Error)

Results of validate.php

===========================================
Component | Version
--------- | -------
LibreNMS  | 22.12.0-18-g4a97c67b9 (2023-01-18T09:46:36-08:00)
DB Schema | 2022_08_15_084507_add_rrd_type_to_wireless_sensors_table (248)
PHP       | 8.1.10
Python    | 3.6.8
Database  | MariaDB 10.5.17-MariaDB
RRDTool   | 1.4.8
SNMP      | 5.7.2
===========================================

[OK]    Composer Version: 2.5.1
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]
[OK]    Database schema correct
[OK]    MySQl and PHP time match
[OK]    Active pollers found
[OK]    Dispatcher Service not detected
[OK]    Locks are functional
[OK]    Python poller wrapper is polling
[OK]    Redis is unavailable
[OK]    rrdtool version ok
[OK]    Connected to rrdcached

Is there some fix for the Microsoft Teams Transport?

I created the webhook for my channel, I tried to set it with json and without json, it always brings me this error:

image

When I use the test-alert.php to debug, it shows me the message below:

:: Transport msteams => SQL[select * from `alert_transports` where `alert_transports`.`transport_id` = ? limit 1 [9] 0.24ms]

string(43) "Microsoft Teams returned Error, retry later"
ERRORSQL[insert into `eventlog` (`reference`, `type`, `datetime`, `severity`, `message`, `username`, `device_id`) values (?, ?, ?, ?, ?, ?, ?) [null,null,"2023-01-18 15:51:24",5,"Could not issue critical alert for rule 'TEST' to transport 'msteams'","",1] 0.22ms]

When I use curl, it works:

curl -H 'Content-Type: application/json' -d '{"text": "Hello World"}' https://xxxxxxxxx.webhook.office.com/webhookb2/xxxxxxxxxxxxxxxx-xxxxxxxxxx@xxxxxxxxxxxx-xxxxxxxxxxxxxxxxx/IncomingWebhook/xxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxxx

Does anyone know how to fix it?

Any help is appreciated.

Thank you.

Works fine for me, though I can recreate your error if I enable JSON, does it work if you set Use JSON to OFF?

image

I vaguely remember someone on here trying to debug that a while back, but clearly it’s still buggy.

Unfortunately no,

I have tried both with and without the JSON flag.

Thank you for your reply though.

I tried to add some more debugging to the transport to compare what happens between working and not, but I see no difference in the headers etc. maybe you can dump out some stuff like this and get the same outputs from native curl and see if you can spot anything amiss.

I’m not a PHP native, but I took a backup of /opt/librenms/LibreNMS/Alert/Transport/Msteams.php and added some header opts and dumped some things out - use it with the test-alert script with -d option and see what you get.

I’ve indented them a lot and add <<=== at the end to show them:

    public function contactMsteams($obj, $opts)
    {
        $url = $opts['url'];
        $data = [
            'title' => $obj['title'],
            'themeColor' => self::getColorForState($obj['state']),
            'text' => strip_tags($obj['msg'], '<strong><em><h1><h2><h3><strike><ul><ol><li><pre><blockquote><a><img><p>'),
            'summary' => $obj['title'],
        ];
        $curl = curl_init();
        Proxy::applyToCurl($curl);
        curl_setopt($curl, CURLOPT_URL, $url);
        curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($curl, CURLOPT_HTTPHEADER, [
            'Content-type' => 'application/json',
            'Expect:',
    ]);
                        curl_setopt($curl, CURLOPT_VERBOSE, 1);                    <<===
                        curl_setopt($curl, CURLOPT_HEADER, 1);                     <<===
        curl_setopt($curl, CURLOPT_POSTFIELDS, json_encode($data));

                        var_dump(json_encode($data));                              <<===
        if ($this->config['use-json'] === 'on' && $obj['uid'] !== '000') {
            curl_setopt($curl, CURLOPT_POSTFIELDS, $obj['msg']);
        }
        $ret = curl_exec($curl);
        $code = curl_getinfo($curl, CURLINFO_HTTP_CODE);
        if ($code != 200) {
            var_dump('Microsoft Teams returned Error, retry later');

                        $header_size = curl_getinfo($curl, CURLINFO_HEADER_SIZE);  <<===
                        $header = substr($ret, 0, $header_size);                   <<===
                        $body = substr($ret, $header_size);                        <<===
                        var_dump($header);                                         <<===
                        var_dump($body);                                           <<===

            return false;
        }

                        $header_size = curl_getinfo($curl, CURLINFO_HEADER_SIZE);  <<===
                        $header = substr($ret, 0, $header_size);                   <<===
                        $body = substr($ret, $header_size);                        <<===
                        var_dump($header);                                         <<===
                        var_dump($body);                                           <<===

        return true;
    }

For me it it dumps out:

This is what I get:

I don’t know if that’s related to the fact we are GCC (Government) tenant.

I don’t know if there’s another header missing.

string(456) "{"title":"DEVICE DOWN","themeColor":"#ff0000","text":"Device Name: librenms.xxx.xxx.xxx.xxx\nDevice Ip Address: 127.0.0.1\nDEVICE DOWN\nSeverity: critical\nTimestamp: 2023-01-19 00:41:07\nUnique-ID: 6263\nRule:  TEST  Faults:\n  #1: sysObjectID = .1.3.6.1.4.1.8072.3.2.10; sysDescr = Linux librenms.xxx.xxx.xxx 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022 x86_64; location_id = 1; \nAlert sent to:\n","summary":"DEVICE DOWN"}"
* About to connect() to xxxxxxxxxxx.webhook.office.com port 443 (#0)
*   Trying 52.96.166.242...
* Connected to xxxxxxx.webhook.office.com (52.96.166.242) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=*.webhook.office.com,O=Microsoft Corporation,L=Redmond,ST=Washington,C=US
*       start date: Aug 26 00:00:00 2022 GMT
*       expire date: Aug 25 23:59:59 2023 GMT
*       common name: *.webhook.office.com
*       issuer: CN=DigiCert Cloud Services CA-1,O=DigiCert Inc,C=US
> POST /webhookb2/7xxxxxxxxxxxxxxxxxxxxxxxx@xxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxx/IncomingWebhook/xxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx HTTP/1.1
Host: xxxxxxxx.webhook.office.com
Accept: */*
Content-Length: 456
Content-Type: application/x-www-form-urlencoded

* upload completely sent off: 456 out of 456 bytes
< HTTP/1.1 400
< Content-Length: 49
< Content-Type: text/plain; charset=utf-8
< Server: Microsoft-IIS/10.0
< request-id: xxxxxxxxxxxxxxxxxxxxx
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Alt-Svc: h3=":443",h3-29=":443"
< X-PreferredRoutingKeyDiagnostics: 1
< X-CalculatedFETarget: xxxxxxxxxxx.internal.outlook.com
< X-BackEndHttpStatus: 400
< X-CalculatedBETarget: xxxxxxxxxxxxxxx.NAMPRD09.PROD.OUTLOOK.COM
< X-BackEndHttpStatus: 400
< X-End2EndLatencyMs: 1329
< X-RUM-Validated: 1
< X-RUM-NotUpdateQueriedPath: 1
< X-Proxy-RoutingCorrectness: 1
< X-MS-ForwardingCorrelationId: xxxxxxxxxxxxxxxxx
< X-Proxy-BackendServerStatus: 400
< X-BEPartition: xxxxxxxxxxxxxxxxxx
< X-FEProxyInfo: xxxxxxxxxxxxxxxxxxxxx.NAMPRD05.PROD.OUTLOOK.COM
< X-FEEFZInfo: SJC
< MS-CV: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
< X-FEServer: SA1P222CA0014
< X-FirstHopCafeEFZ: SJC
< X-Powered-By: ASP.NET
< X-FEServer: xxxxxxxxxxxxxxxxxxxxxxxx
< Date: Thu, 19 Jan 2023 09:00:14 GMT
<
* Connection #0 to host xxxxxxxxxxxxxxxx.webhook.office.com left intact
string(43) "Microsoft Teams returned Error, retry later"
string(960) "HTTP/1.1 400
Content-Length: 49
Content-Type: text/plain; charset=utf-8
Server: Microsoft-IIS/10.0
request-id: 2f3bfee0-6c0a-6858-55e2-1d94d3405fa1
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Alt-Svc: h3=":443",h3-29=":443"
X-PreferredRoutingKeyDiagnostics: 1
X-CalculatedFETarget: xxxxxxxxxxxxx.internal.outlook.com
X-BackEndHttpStatus: 400
X-CalculatedBETarget: xxxxxxxxxxxxxx.NAMPRD09.PROD.OUTLOOK.COM
X-BackEndHttpStatus: 400
X-End2EndLatencyMs: 1329
X-RUM-Validated: 1
X-RUM-NotUpdateQueriedPath: 1
X-Proxy-RoutingCorrectness: 1
X-MS-ForwardingCorrelationId: xxxxxxxxxxxxxxxxx
X-Proxy-BackendServerStatus: 400
X-BEPartition: xxxxxxxxxxxxxx
X-FEProxyInfo: xxxxxxxxxxxxxxxxxxxx.NAMPRD05.PROD.OUTLOOK.COM
X-FEEFZInfo: SJC
MS-CV: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
X-FEServer: SA1P222CA0014
X-FirstHopCafeEFZ: SJC
X-Powered-By: ASP.NET
X-FEServer: SJ0PR05CA0039
Date: Thu, 19 Jan 2023 09:00:14 GMT

string(49) "Bad payload received by generic incoming webhook."
ERRORSQL[insert into `eventlog` (`reference`, `type`, `datetime`, `severity`, `message`, `username`, `device_id`) values (?, ?, ?, ?, ?, ?, ?) [null,null,"2023-01-19 01:16:02",5,"Could not issue critical alert for rule 'TEST' to transport 'msteams'","",1] 0.18ms]

I was able to intercept the connection using the proxy of the Burp Suite Software:

This is the payload I get with the json flag on:

POST /webhookb2/XXXXXXXXXXXXXXXXXXXXX@cXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXXXXXXX/IncomingWebhook/XXXXXXXXXXXXXXXXXXXXXXX/XXXXXXXXXXXXXXXXXXXX HTTP/2

Host: XXXXXXXXXXXXX.webhook.office.com

Accept: */*

Content-Length: 1728

Content-Type: application/x-www-form-urlencoded



{
    "@context": "https://schema.org/extensions",
    "@type": "MessageCard",
    "title": "Device UP",
    "themeColor": "FF0000",
    "summary": "LibreNMS",
    "sections": [
        {
            "facts": [
                {
                    "name": "Rule:",
                    "value": "[Test 2](https://your.librenms.url/device/device=1/tab=alert/)"
                },
                {
                    "name": "Severity:",
                    "value": "critical"
                },
                {
                    "name": "Unique-ID:",
                    "value": "6314"
                },
                {
                    "name": "Timestamp:",
                    "value": "2023-01-19 10:05:25"
                },
                {
                    "name": "Hostname:",
                    "value": "[127.0.0.1](https://your.librenms.url/device/device=1/)"
                },
                {
                    "name": "Hardware:",
                    "value": "Generic x86 64-bit"
                },
                {
                    "name": "IP:",
                    "value": ""
                },
                {
                    "name": "Faults:",
                    "value": " "
                }
            ]
        },
        {
            "facts": [
                {
                    "name": "Port:",
                    "value": "[](https://your.librenms.url/device/device=1/tab=port/port=/)"
                },
                {
                    "name": "Description:",
                    "value": ""
                },
                {
                    "name": "Status:",
                    "value": "down"
                }
            ]
        }
    ]
}

This is the payload I get using the json flag off:

POST /webhookb2/XXXXXXXXXXXXXXXXXXXX@XXXXXXXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXXXX/IncomingWebhook/XXXXXXXXXXXXXXXXXXXXXXX/XXXXXXXXXXXXXXXXXXXXX HTTP/2

Host: XXXXXXXXXXX.webhook.office.com

Accept: */*

Content-Length: 2000

Content-Type: application/x-www-form-urlencoded



{"title":"Device UP","themeColor":"#ff0000","text":"{\n    \"@context\": \"https:\/\/schema.org\/extensions\",\n    \"@type\": \"MessageCard\",\n    \"title\": \"Device UP\",\n    \"themeColor\": \"FF0000\",\n    \"summary\": \"LibreNMS\",\n    \"sections\": [\n        {\n            \"facts\": [\n                {\n                    \"name\": \"Rule:\",\n                    \"value\": \"[Test 2](https:\/\/your.librenms.url\/device\/device=1\/tab=alert\/)\"\n                },\n                {\n                    \"name\": \"Severity:\",\n                    \"value\": \"critical\"\n                },\n                {\n                    \"name\": \"Unique-ID:\",\n                    \"value\": \"6314\"\n                },\n                {\n                    \"name\": \"Timestamp:\",\n                    \"value\": \"2023-01-19 10:05:25\"\n                },\n                {\n                    \"name\": \"Hostname:\",\n                    \"value\": \"[127.0.0.1](https:\/\/your.librenms.url\/device\/device=1\/)\"\n                },\n                {\n                    \"name\": \"Hardware:\",\n                    \"value\": \"Generic x86 64-bit\"\n                },\n                {\n                    \"name\": \"IP:\",\n                    \"value\": \"\"\n                },\n                {\n                    \"name\": \"Faults:\",\n                    \"value\": \" \"\n                }\n            ]\n        },\n        {\n            \"facts\": [\n                {\n                    \"name\": \"Port:\",\n                    \"value\": \"[](https:\/\/your.librenms.url\/device\/device=1\/tab=port\/port=\/)\"\n                },\n                {\n                    \"name\": \"Description:\",\n                    \"value\": \"\"\n                },\n                {\n                    \"name\": \"Status:\",\n                    \"value\": \"down\"\n                }\n            ]\n        }\n    ]\n}\n","summary":"Device UP"}

For both of them I get the same error:

string(456) "{"title":"DEVICE DOWN","themeColor":"#ff0000","text":"Device Name: librenms.xxx.xxx.xxx.xxx\nDevice Ip Address: 127.0.0.1\nDEVICE DOWN\nSeverity: critical\nTimestamp: 2023-01-19 00:41:07\nUnique-ID: 6263\nRule:  TEST  Faults:\n  #1: sysObjectID = .1.3.6.1.4.1.8072.3.2.10; sysDescr = Linux librenms.xxx.xxx.xxx 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022 x86_64; location_id = 1; \nAlert sent to:\n","summary":"DEVICE DOWN"}"
* About to connect() to xxxxxxxxxxx.webhook.office.com port 443 (#0)
*   Trying 52.96.166.242...
* Connected to xxxxxxx.webhook.office.com (52.96.166.242) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=*.webhook.office.com,O=Microsoft Corporation,L=Redmond,ST=Washington,C=US
*       start date: Aug 26 00:00:00 2022 GMT
*       expire date: Aug 25 23:59:59 2023 GMT
*       common name: *.webhook.office.com
*       issuer: CN=DigiCert Cloud Services CA-1,O=DigiCert Inc,C=US
> POST /webhookb2/7xxxxxxxxxxxxxxxxxxxxxxxx@xxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxx/IncomingWebhook/xxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx HTTP/1.1
Host: xxxxxxxx.webhook.office.com
Accept: */*
Content-Length: 456
Content-Type: application/x-www-form-urlencoded

* upload completely sent off: 456 out of 456 bytes
< HTTP/1.1 400
< Content-Length: 49
< Content-Type: text/plain; charset=utf-8
< Server: Microsoft-IIS/10.0
< request-id: xxxxxxxxxxxxxxxxxxxxx
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Alt-Svc: h3=":443",h3-29=":443"
< X-PreferredRoutingKeyDiagnostics: 1
< X-CalculatedFETarget: xxxxxxxxxxx.internal.outlook.com
< X-BackEndHttpStatus: 400
< X-CalculatedBETarget: xxxxxxxxxxxxxxx.NAMPRD09.PROD.OUTLOOK.COM
< X-BackEndHttpStatus: 400
< X-End2EndLatencyMs: 1329
< X-RUM-Validated: 1
< X-RUM-NotUpdateQueriedPath: 1
< X-Proxy-RoutingCorrectness: 1
< X-MS-ForwardingCorrelationId: xxxxxxxxxxxxxxxxx
< X-Proxy-BackendServerStatus: 400
< X-BEPartition: xxxxxxxxxxxxxxxxxx
< X-FEProxyInfo: xxxxxxxxxxxxxxxxxxxxx.NAMPRD05.PROD.OUTLOOK.COM
< X-FEEFZInfo: SJC
< MS-CV: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
< X-FEServer: SA1P222CA0014
< X-FirstHopCafeEFZ: SJC
< X-Powered-By: ASP.NET
< X-FEServer: xxxxxxxxxxxxxxxxxxxxxxxx
< Date: Thu, 19 Jan 2023 09:00:14 GMT
<
* Connection #0 to host xxxxxxxxxxxxxxxx.webhook.office.com left intact
string(43) "Microsoft Teams returned Error, retry later"
string(960) "HTTP/1.1 400
Content-Length: 49
Content-Type: text/plain; charset=utf-8
Server: Microsoft-IIS/10.0
request-id: 2f3bfee0-6c0a-6858-55e2-1d94d3405fa1
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Alt-Svc: h3=":443",h3-29=":443"
X-PreferredRoutingKeyDiagnostics: 1
X-CalculatedFETarget: xxxxxxxxxxxxx.internal.outlook.com
X-BackEndHttpStatus: 400
X-CalculatedBETarget: xxxxxxxxxxxxxx.NAMPRD09.PROD.OUTLOOK.COM
X-BackEndHttpStatus: 400
X-End2EndLatencyMs: 1329
X-RUM-Validated: 1
X-RUM-NotUpdateQueriedPath: 1
X-Proxy-RoutingCorrectness: 1
X-MS-ForwardingCorrelationId: xxxxxxxxxxxxxxxxx
X-Proxy-BackendServerStatus: 400
X-BEPartition: xxxxxxxxxxxxxx
X-FEProxyInfo: xxxxxxxxxxxxxxxxxxxx.NAMPRD05.PROD.OUTLOOK.COM
X-FEEFZInfo: SJC
MS-CV: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
X-FEServer: SA1P222CA0014
X-FirstHopCafeEFZ: SJC
X-Powered-By: ASP.NET
X-FEServer: SJ0PR05CA0039
Date: Thu, 19 Jan 2023 09:00:14 GMT

string(49) "Bad payload received by generic incoming webhook."
ERRORSQL[insert into `eventlog` (`reference`, `type`, `datetime`, `severity`, `message`, `username`, `device_id`) values (?, ?, ?, ?, ?, ?, ?) [null,null,"2023-01-19 01:16:02",5,"Could not issue critical alert for rule 'TEST' to transport 'msteams'","",1] 0.18ms]

Now if I use postman to make the same two request, it works:

Json flag on
Postman

Teams

Json flag off

PostMan

Teams

Maybe there is a header missing or something like that, I am not a programmer so I can’t tell :sleepy:

Thank you

Hi,

We realized that we are experiencing the same situation in 10-15 different places, and we are looking for a solution. We can create notifications with powershell and curl, but the php script is suddenly broken.

 :: Transport msteams => SQL[select * from `alert_transports` where `alert_transports`.`transport_id` = ? limit 1 [2] 0.27ms] 
  
string(43) "Microsoft Teams returned Error, retry later"
string(49) "Bad payload received by generic incoming webhook."
string(0) ""
ERRORSQL[insert into `eventlog` (`reference`, `type`, `datetime`, `severity`, `message`, `username`, `device_id`) values (?, ?, ?, ?, ?, ?, ?) [null,null,"2023-01-19 22:30:41",5,"Could not issue critical alert for rule 'Cihaz Erisim Hatasi' to transport 'msteams'","",471] 0.38ms] 

Thanks.

@kendinitekmeleyenked I think I found the solution:

On the file: /opt/librenms/LibreNMS/Alert/Transport/Msteams.php (Assuming that’s the path for your MSteams.php file)

Where is:

curl_setopt($curl, CURLOPT_HTTPHEADER, [
            'Content-type' => 'application/json',
            'Expect:',
        ]);

Change it to:

curl_setopt($curl, CURLOPT_HTTPHEADER, array('Content-Type:application/json', 'Expect:'));

I just got it working.

Let me know if it worked for you.

3 Likes

Senin canini yerim <3
I’ll eat your soul <3

1 Like

Hi @michelbragaguimaraes,

You are great, thank youuuu. :slight_smile:

Thank you, my friend.

Nice work! It causes no change on mine, it continues to work with ‘Use JSON’ off, and still fails with it on - which is more to do with how it constructs that JSON I suppose. Does leaving JSON on work for you?

Unsure why yours never works to start with and mine did though. I don’t put much time in to teams, as it can’t keep up on mobile with the alert volume and simply falls over, so I can’t depend on it and moved to slack for alerting.

It works with both Json on or off.

Now my alerts are working as it should

I are changing from slack to teams effectively.

My slack alerts always worked flawlessly.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.