Lots of TLS log errors, but everything working

[leonnortje]

I've setup a Teleport cluster with nodes connecting via reverse tunnel.

Most functionality is working: I can access the web interface, add nodes, individual nodes show up correctly as <- tunnel nodes, and can connect to them via the web interface, and connect to them via tsh command line tool. Great!

However, there is a persistent TLS log error, which happens on the proxy, per connected node.

The error happens seemingly on a back-off schedule: after 1 second, 1 second, 2 seconds, 4 seconds, 6, 10, etc.

When we use our own valid wildcard certificate:

ERRO [PROXY:SER] "proxy2021/03/21 14:18:11 http: TLS handshake error from 1.2.3.4:25844: remote error: tls: bad certificate\n" utils/cli.go:287

This is a very unhelpful error!

Switching over to the ACME service it gets a bit more useful:

ERRO [PROXY:SER] "proxy2021/03/19 16:45:56 http: TLS handshake error from 1.2.3.4:32120: acme can't get a cert for domain 74656c65706f72742e696f6e6f2e7161.teleport.cluster.local, add it to the proxy_service.public_addr, or use one of the domains: HOST.EXAMPLE.COM\n" utils/cli.go:287

(Where HOST.EXAMPLE.COM and 1.2.3.4 replaced the actual values)

Q:

1. Everything seems to be working, what will this error affect ?
2. What is causing this issue ?
3. How do we resolve this ?

I spent 1 hour getting the cluster up and then 2 days trying to get resolve this error. Very frustrating!

At this point I almost just want to filter this error out of our syslogs and call it a day, but i do not really want unexplained errors on our control plane which may come back to bite us one day..

(Teleport can certainly do with much more useful error logging.)