Telegram stopped working

Hi,
Since yesterday my notifications with Telegram stopped working.
When I run a test-alert I get the following output:

/opt/librenms$ sudo -u librenms ./scripts/test-alert.php -r 35 -h 7 -d
SQL[SELECT alerts.id, alerts.device_id, alerts.rule_id, alerts.state, alerts.note, alerts.info FROM alerts WHERE alerts.device_id = 7 && alerts.rule_id = 35 [] 0.64ms]

SQL[SELECT alert_log.id,alert_log.rule_id,alert_log.device_id,alert_log.state,alert_log.details,alert_log.time_logged,alert_rules.rule,alert_rules.severity,alert_rules.extra,alert_rules.name,alert_rules.query,alert_rules.builder,alert_rules.proc FROM alert_log,alert_rules WHERE alert_log.rule_id = alert_rules.id && alert_log.device_id = ? && alert_log.rule_id = ? && alert_rules.disabled = 0 ORDER BY alert_log.id DESC LIMIT 1 [7,35] 0.35ms]

SQL[SELECT DISTINCT a.* FROM alert_rules a
LEFT JOIN alert_device_map d ON a.id=d.rule_id
LEFT JOIN alert_group_map g ON a.id=g.rule_id
LEFT JOIN device_group_device dg ON g.group_id=dg.device_group_id
WHERE a.disabled = 0 AND ((d.device_id IS NULL AND g.group_id IS NULL) OR d.device_id=? OR dg.device_id=?) [7,7] 0.3ms]

SQL[SELECT attrib_value FROM devices_attribs WHERE attrib_type = “disable_notify” && device_id = ? [7] 0.46ms]

SQL[SELECT hostname, sysName, sysDescr, sysContact, os, type, ip, hardware, version, purpose, notes, uptime, status, status_reason, locations.location FROM devices LEFT JOIN locations ON locations.id = devices.location_id WHERE device_id = ? [7] 0.41ms]

SQL[SELECT * FROM devices_attribs WHERE device_id = ? [7] 0.33ms]

SQL[select * from device_perf where device_id = ? order by timestamp desc limit 1 [7] 22.69ms]

SQL[select * from alert_templates where exists (select * from alert_template_map where alert_templates.id = alert_template_map.alert_templates_id and alert_rule_id = ?) limit 1 [35] 0.97ms]

Issuing Alert-UID #14205/1:
SQL[SELECT rule_id FROM alerts WHERE id=? [2477] 0.32ms]

SQL[SELECT b.transport_id, b.transport_type, b.transport_name FROM alert_transport_map AS a LEFT JOIN alert_transports AS b ON b.transport_id=a.transport_or_group_id WHERE a.target_type=‘single’ AND a.rule_id=? UNION DISTINCT SELECT d.transport_id, d.transport_type, d.transport_name FROM alert_transport_map AS a LEFT JOIN alert_transport_groups AS b ON a.transport_or_group_id=b.transport_group_id LEFT JOIN transport_group_transport AS c ON b.transport_group_id=c.transport_group_id LEFT JOIN alert_transports AS d ON c.transport_id=d.transport_id WHERE a.target_type=‘group’ AND a.rule_id=? [35,35] 0.91ms]

:: Transport telegram => SQL[SELECT transport_config FROM alert_transports WHERE transport_id=? [2] 0.28ms]

string(21) “API ‘’ returned Error”
string(8) “Params: "
string(8) “Return: "
ERROR: HTTP Status code 0
SQL[insert into eventlog (device_id, reference, type, datetime, severity, message, username) values (?, ?, ?, ?, ?, ?, ?) [7,null,“error”,“2019-11-14 14:31:49”,5,“Could not issue critical alert for rule ‘Device rebooted’ to transport ‘telegram’ Error: HTTP Status code 0”,””] 22.59ms]

During it times out after 5 minutes on “:: Transport telegram =>” and then finally returns the error “HTTP Status code 0”

What I tried so far is to reset permissions with:
chown -R librenms:librenms /opt/librenms/storage/
chmod -R g+w /opt/librenms/storage

I have updated LibreNMS to the latest version

And created a new Telegram transport.
When I hit the test transport button the (failure) notification also shows after a 5 minute time-out.

What dose ./validate.php show?

Here is the output:

/opt/librenms$ sudo -u librenms ./validate.php

====================================

Component Version
LibreNMS 1.57-52-g4b0c77515
DB Schema 2019_08_10_223200_add_enabled_to_users (146)
PHP 7.2.24-0ubuntu0.18.04.1
MySQL 10.1.41-MariaDB-0ubuntu0.18.04.1
RRDTool 1.7.0
SNMP NET-SNMP 5.7.3

====================================

[OK] Composer Version: 1.9.1
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database schema correct

/opt/librenms$ sudo -u librenms ./daily.sh

Updating to latest codebase OK
Updating Composer packages OK
Updating SQL-Schema OK
Updating submodules OK
Cleaning up DB OK
Fetching notifications OK
Caching PeeringDB data OK

Strange thing is I don’t see any failures.
After the latest upgrade and validate I did a reboot of the server but still no Telegram notifications.
Just tested a mail transport, that does work.

It looks like notifications are being sent inconsistently. I have enabled mail notifications, but they don’t always work either. Last night I received 1 Telegram notification and a few mail notifications (i should receive all notifications both on mail and Telegram). Some notifications I get a recovery message from but others do not.
Could it perhaps not be a capacity thing?
The poller takes about 3 minutes.

Check time zones in php and the system time on the serves that’s running librenms.

looks like https://community.librenms.org/t/http-error-code-0-in-telegram

Dont know if that user fixed it, but for sure I think theres nothing to do with LibreNMS, as mine works and send notifications to telegram as expected.

System and PHP timezone are both the same and set correct. The system time is also correct.

I checked the other forum post. When i google for HTTP error code 0 it looks like the cause is a problem with no feedback from the API. That might explain the timeout is notice after testing the notification.

I also see comparable malfunctions where an error in the default template is the cause, which I have just adjusted.

Okay, a little progression.

Checked the connection to telegram:
:~$ curl -v https://api.telegram.org

  • Rebuilt URL to: https://api.telegram.org/
  • Trying 149.154.167.220…
  • TCP_NODELAY set
  • Connected to api.telegram.org (149.154.167.220) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: /etc/ssl/certs
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to api.telegram.org:443
  • stopped the pause stream!
  • Closing connection 0
    curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to api.telegram.org:443

Also checked the installed version of curl:
:~$ curl -V
curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

When I google the message “curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to api.telegram.org:443” it seems like an firewall issue. Have to check my network again…

For anyone with the same issues, you can send your own Telegram message from the command line with the following command:
curl -X POST “https://api.telegram.org/bot<BOT_TOKEN>/sendMessage” -d “chat_id=<CHAT_ID>&text=my sample text”

I currently have a workaround for the issue. Seems to be something in our network.
After checking with wireshark it seemed to be an MTU issue. I changed the MTU of the network interface and Telegram is working again!