Changing nodename on cluster root causes failed auth

[dudeisbrendan03]

After changing the nodename on our cluster root, all leafs and users can no longer connect to the cluster. The web interface works but that's about it.

We renamed it from "001.xyz.eu" to "001.lndn.eu" and it no longer appears to function.

We've restarted the service, attempted to re-authenticate and restarted leaves, nothing.

error: failed to authenticate with proxy teleport.abc.xyz:3023: ssh: handshake failed: ssh: principal "teleport.abc.xyz" not in the set of valid principals for given certificate: ["169dd3d7-3f28-41cb-858e-ed33ce09db09.TPC-ManageNet" "169dd3d7-3f28-41cb-858e-ed33ce09db09" "localhost" "127.0.0.1" "::1" "001.lndn.eu" "remote.kube.proxy.teleport.cluster.local"]

Running 6.0.1, didn't modify the config at all apart from the nodename but I can publish it.

No errors or warnings when someone tries to login, only ones on the client side.

Client debug:

ERRO [KEYSTORE]  open /home/brendan/.tsh/keys/teleport.abc.xyz/brendantdj-cert.pub: no such file or directory client/keystore.go:219
INFO [CLIENT]    no host login given. defaulting to brendan client/api.go:801
ERRO [CLIENT]    [KEY AGENT] Unable to connect to SSH agent on socket: "". client/api.go:2225
ERRO [KEYSTORE]  open /home/brendan/.tsh/keys/teleport.abc.xyz/brendantdj-cert.pub: no such file or directory client/keystore.go:219
DEBU [CLIENT]    not using loopback pool for remote proxy addr: teleport.abc.xyz:3080 client/api.go:2190
DEBU [CLIENT]    HTTPS client init(proxyAddr=teleport.abc.xyz:3080, insecure=false) client/weblogin.go:307
Enter password for Teleport user brendantdj:
Enter your OTP token:
686162
DEBU [CLIENT]    not using loopback pool for remote proxy addr: teleport.abc.xyz:3080 client/api.go:2190
DEBU [CLIENT]    HTTPS client init(proxyAddr=teleport.abc.xyz:3080, insecure=false) client/weblogin.go:307
DEBU [KEYAGENT]  Adding CA key for TPC-ManageNet client/keyagent.go:243
DEBU [KEYSTORE]  Adding known host TPC-ManageNet with key: SHA256:b4h0odrirQGuggNJwKkti9lpQ+Q0fT0sfuivCPj23uo client/keystore.go:381
INFO [CLIENT]    Connecting proxy=teleport.abc.xyz:3023 login='administrator' method=0 client/api.go:1641
DEBU [KEYAGENT]  Host validation failed: ssh: principal "teleport.abc.xyz" not in the set of valid principals for given certificate: ["169dd3d7-3f28-41cb-858e-ed33ce09db09.TPC-ManageNet" "169dd3d7-3f28-41cb-858e-ed33ce09db09" "localhost" "127.0.0.1" "::1" "001.lndn.eu" "remote.kube.proxy.teleport.cluster.local"]. client/keyagent.go:282
WARN [CLIENT]    Failed to authenticate with proxy: ssh: handshake failed: ssh: principal "teleport.abc.xyz" not in the set of valid principals for given certificate: ["169dd3d7-3f28-41cb-858e-ed33ce09db09.TPC-ManageNet" "169dd3d7-3f28-41cb-858e-ed33ce09db09" "localhost" "127.0.0.1" "::1" "001.lndn.eu" "remote.kube.proxy.teleport.cluster.local"] client/api.go:1644
INFO [CLIENT]    Connecting proxy=teleport.abc.xyz:3023 login='administrator' method=0 client/api.go:1641
DEBU [KEYAGENT]  Host validation failed: ssh: principal "teleport.abc.xyz" not in the set of valid principals for given certificate: ["169dd3d7-3f28-41cb-858e-ed33ce09db09.TPC-ManageNet" "169dd3d7-3f28-41cb-858e-ed33ce09db09" "localhost" "127.0.0.1" "::1" "001.lndn.eu" "remote.kube.proxy.teleport.cluster.local"]. client/keyagent.go:282
WARN [CLIENT]    Failed to authenticate with proxy: ssh: handshake failed: ssh: principal "teleport.abc.xyz" not in the set of valid principals for given certificate: ["169dd3d7-3f28-41cb-858e-ed33ce09db09.TPC-ManageNet" "169dd3d7-3f28-41cb-858e-ed33ce09db09" "localhost" "127.0.0.1" "::1" "001.lndn.eu" "remote.kube.proxy.teleport.cluster.local"] client/api.go:1644

ERROR REPORT:
Original Error: *trace.BadParameterError failed to authenticate with proxy teleport.abc.xyz:3023: ssh: handshake failed: ssh: principal "teleport.abc.xyz" not in the set of valid principals for given certificate: ["169dd3d7-3f28-41cb-858e-ed33ce09db09.TPC-ManageNet" "169dd3d7-3f28-41cb-858e-ed33ce09db09" "localhost" "127.0.0.1" "::1" "001.lndn.eu" "remote.kube.proxy.teleport.cluster.local"]
Stack Trace:
        /go/src/github.com/gravitational/teleport/lib/client/api.go:1637 github.com/gravitational/teleport/lib/client.(*TeleportClient).connectToProxy
        /go/src/github.com/gravitational/teleport/lib/client/api.go:1591 github.com/gravitational/teleport/lib/client.(*TeleportClient).ConnectToProxy.func1
        /opt/go/src/runtime/asm_amd64.s:1374 runtime.goexit
User Message: failed to authenticate with proxy teleport.abc.xyz:3023: ssh: handshake failed: ssh: principal "teleport.abc.xyz" not in the set of valid principals for given certificate: ["169dd3d7-3f28-41cb-858e-ed33ce09db09.TPC-ManageNet" "169dd3d7-3f28-41cb-858e-ed33ce09db09" "localhost" "127.0.0.1" "::1" "001.lndn.eu" "remote.kube.proxy.teleport.cluster.local"]

Server debug:

DEBU [AUTH:1]    Server certificate cert(169dd3d7-3f28-41cb-858e-ed33ce09db09.TPC-ManageNet issued by TPC-ManageNet:73976220968790758829608208191109068758). auth/middleware.go:286
INFO [AUDIT]     user.login cluster_name: code:T1000I ei:0 event:user.login method:local success:true time:2021-03-16T19:30:41.464Z uid:680c14c2-5fff-4d35-a558-f742aa8b6e10 user:brendantd7
DEBU [KEYGEN]    generated user key for [administrator tpc-ext tpc-int] with expiry on (1615966241) 2021-03-17 07:30:41.471336286 +0000 UTC native/native.go:257
DEBU [AUTH]      Failed setting default kubernetes cluster for user login (user did not provide a cluster); leaving KubernetesCluster extension in the TLS certificate empty error:[
ERROR REPORT:
Original Error: *trace.NotFoundError no kubernetes clusters registered
Stack Trace:
        /go/src/github.com/gravitational/teleport/lib/kube/utils/utils.go:192 github.com/gravitational/teleport/lib/kube/utils.CheckOrSetKubeCluster
        /go/src/github.com/gravitational/teleport/lib/auth/auth.go:708 github.com/gravitational/teleport/lib/auth.(*Server).generateUserCert
        /go/src/github.com/gravitational/teleport/lib/auth/methods.go:364 github.com/gravitational/teleport/lib/auth.(*Server).AuthenticateSSHUser
        /go/src/github.com/gravitational/teleport/lib/auth/auth_with_roles.go:150 github.com/gravitational/teleport/lib/auth.(*ServerWithRoles).AuthenticateSSHUser
        /go/src/github.com/gravitational/teleport/lib/auth/apiserver.go:801 github.com/gravitational/teleport/lib/auth.(*APIServer).authenticateSSHUser
        /go/src/github.com/gravitational/teleport/lib/auth/apiserver.go:292 github.com/gravitational/teleport/lib/auth.(*APIServer).withAuth.func1
        /go/src/github.com/gravitational/teleport/lib/httplib/httplib.go:61 github.com/gravitational/teleport/lib/httplib.MakeHandlerWithErrorWriter.func1
        /go/src/github.com/gravitational/teleport/vendor/github.com/julienschmidt/httprouter/router.go:387 github.com/julienschmidt/httprouter.(*Router).ServeHTTP
        /go/src/github.com/gravitational/teleport/lib/httplib/httplib.go:173 github.com/gravitational/teleport/lib/httplib.RewritePaths.func1
        /opt/go/src/net/http/server.go:2042 net/http.HandlerFunc.ServeHTTP
        /go/src/github.com/gravitational/teleport/lib/auth/middleware.go:529 github.com/gravitational/teleport/lib/auth.(*Middleware).ServeHTTP
        /go/src/github.com/gravitational/teleport/vendor/github.com/gravitational/oxy/ratelimit/tokenlimiter.go:118 github.com/gravitational/oxy/ratelimit.(*TokenLimiter).ServeHTTP
        /go/src/github.com/gravitational/teleport/vendor/github.com/gravitational/oxy/connlimit/connlimit.go:75 github.com/gravitational/oxy/connlimit.(*ConnLimiter).ServeHTTP
        /opt/go/src/net/http/server.go:2843 net/http.serverHandler.ServeHTTP
        /opt/go/src/net/http/server.go:1925 net/http.(*conn).serve
        /opt/go/src/runtime/asm_amd64.s:1374 runtime.goexit
User Message: no kubernetes clusters registered] auth/auth.go:713
INFO [CA]        Generating TLS certificate {0x452f9d8 0xc000a1cbf0 1.3.9999.1.7=#130d5450432d4d616e6167654e6574,1.3.9999.1.1=#130a6272656e64616e74646a,CN=brendantdj,O=admin,POSTALCODE={\2
DEBU [RBAC]      Check access to role(Proxy) rc(Client- , labels=map[]) matchLabels=true, msg=matched, err=<nil> allow=map[*:[*]] rcLabels=map[] services/role.go:1307
DEBU [RBAC]      Check access to role(Proxy) rc(Client- , labels=map[]) matchLabels=true, msg=matched, err=<nil> allow=map[*:[*]] rcLabels=map[] services/role.go:1307
DEBU [RBAC]      Check access to role(Proxy) rc(TPC- , labels=map[]) matchLabels=true, msg=matched, err=<nil> allow=map[*:[*]] rcLabels=map[] services/role.go:1307
DEBU [RBAC]      Check access to role(Proxy) rc(TPC- , labels=map[]) matchLabels=true, msg=matched, err=<nil> allow=map[*:[*]] rcLabels=map[] services/role.go:1307
DEBU [RBAC]      Check access to role(Proxy) rc(Client- , labels=map[]) matchLabels=true, msg=matched, err=<nil> allow=map[*:[*]] rcLabels=map[] services/role.go:1307
DEBU [RBAC]      Check access to role(Proxy) rc(Client- , labels=map[]) matchLabels=true, msg=matched, err=<nil> allow=map[*:[*]] rcLabels=map[] services/role.go:1307
DEBU [RBAC]      Check access to role(Proxy) rc(TPC- , labels=map[]) matchLabels=true, msg=matched, err=<nil> allow=map[*:[*]] rcLabels=map[] services/role.go:1307
DEBU [RBAC]      Check access to role(Proxy) rc(TPC- , labels=map[]) matchLabels=true, msg=matched, err=<nil> allow=map[*:[*]] rcLabels=map[] services/role.go:1307

Only error on the server is related to Kubernetes, this environment does not use nor have kubernetes configured on Teleport.

Edit: leafs are authenticating with our root cluster, although cli (tsh) still wont work- tried clearing out ~/.tsh

[dudeisbrendan03]

public_addr was commented out and I didn't notice, my bad