old offline appliances connecting to updated master

[psuter80]

I wanted to setup teleport for remotely supporting appliances which only connect to the proxy upon request, if assistance is needed. Basically the user of the appliance can initiate the connection to allow support to help him with an issue.

this works perfectly well with the appliances being configured as nodes of a teleport cluster and using revese-tunnels for those appliances (as they are often placed behind firewalls and what not).

however, one issue i have discovered during my evaluation is updates and backward compatibility. If I understand the admin manual correctly, I can't run a proxy version 8.x in the future and an appliance which still may have the version 6.x teleport daemon on it which was installed today can connect to it, as it is more than one major release behind. So I would need to update before connecting. However, as far as I understood, "offline" updating a component such as a node which is more than one major release behind the proxy is not possible because updates are only possible from one major release to another. So if I wanted to upgrade my proxy, auth and tsh clients to a new major release, i would need to bring all my appliances online to upgrade them as well, before i can move to the next major release in the future.

Is my understanding correct?

If so, i see no way to make teleport work for my use case. I could probably implement some kind of "update before connect" mechanism, where the appliance first downloads the latest telepath binary from a trusted server. I could even implement a script which would download major release after major release and launch the binary for a (probably unsuccessful) connection attempt before upgrading to the next version.. however, this would imply that the teleport deamon on the node could update itself without being able to connect to the proxy and i guess that's not a guaranteed or tested behaviour, right?

so the only solution to make this work would be, to let the appliance "call home" on a regular basis to check if updates are needed and then install them.. then wait for all appliances to have checked in before moving to the next major release. in my case, that's not possible because appliances are sometimes very offline and only brought back online manually for support reasons.

cheers
pascal

[webvictim]

Hi Pascal - you're pretty much right with all of this. There is no guarantee that older (6.x) nodes will be able to connect to newer (8.x) proxies.

How many nodes are you thinking about deploying for each customer? For a situation where support is only required very rarely, I would be tempted to just use ephemeral nodes for connection. Basically whenever a customer needs support, you generate a token locally on your cluster with tctl tokens add --ttl=4h --type=node, then give them a command like teleport start --auth-server=teleport.pascal.com --token=bw6mSETOZOL4nqoo3tezB2KpbLh8SvUp --roles=node --labels customer=customername so they can join your cluster on a one-off basis. You could wrap this command in a script which automatically downloads the latest version of Teleport before running, or even use a Docker container if possible so that the customer doesn't have to run anything, they just start Teleport and you provide the support remotely.

For a more resilient solution you might need to do what you described above, i.e. implement a rolling upgrade procedure which will step through every major version. The other thing you should consider is that if you rotate the CA on your Teleport cluster (using tctl auth rotate then the offline nodes would lose connection anyway, as they would not be able to get updated certificates unless they were to come online during the CA rotation.