Fix timeout when defragmenting etcd on startup

[brandond]

Proposed Changes

Fix timeout when defragmenting etcd on startup:

• Use custom client with no hidden background connection, and without automatic retry of RPCs.
• Only timeout status requests; allow defrag and alarm clear requests to run to completion. The defrag and alarm clear may take more than 30 seconds to complete if the datastore is large and the disk is slow.
• Don't clear all alarms on the cluster after defragging, just clear the NOSPACE alarm on the local node.
• Add tests

Types of Changes

bugfix / enhancement

Verification

• See linked issue - however this seems to require a large (8gb+) fragmented datastore on a fairly slow disk to reproduce.

Testing

new tests added

Linked Issues

• Cannot replace etcd nodes over certain sizes #11122
• Bump etcd to 3.5.16 #11171

User-Facing Change

Further Comments

[brandond]

Function was not used here or in rke2 and is not required by the interface.

[codecov]

Codecov Report

Attention: Patch coverage is 59.68992% with 52 lines in your changes missing coverage. Please review.

Project coverage is 42.10%. Comparing base (c0d661b) to head (eede569).
Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/etcd/resolver.go	45.83%	23 Missing and 3 partials ⚠️
pkg/etcd/etcd.go	67.10%	18 Missing and 7 partials ⚠️
pkg/etcd/snapshot.go	0.00%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (c0d661b) and HEAD (eede569). Click for more details.

HEAD has 2 uploads less than BASE

Flag	BASE (c0d661b)	HEAD (eede569)
e2etests	8	6

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #11164      +/-   ##
==========================================
- Coverage   48.72%   42.10%   -6.62%     
==========================================
  Files         178      179       +1     
  Lines       14813    18566    +3753     
==========================================
+ Hits         7218     7818     +600     
- Misses       6288     9540    +3252     
+ Partials     1307     1208      -99     

Flag	Coverage Δ
e2etests	33.93% <0.00%> (-12.13%)	⬇️
inttests	34.69% <44.80%> (?)
unittests	13.58% <58.13%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[manuelbuil]

Shouldn't we add a (larger) timeout for defrag? Could etcd get stuck in defrag forever for some reason?

[brandond]

The reason we added a timeout on the status command is that the etcd grpc client will retry it automatically if the connection to the service is unavailable. I don't believe the other commands are subject to retry, but I'm going to add some tests to this to verify behavior before merging.

[brandond]

Confirmed no breakage here:

NAME           STATUS   ROLES                       AGE     VERSION                INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                   KERNEL-VERSION   CONTAINER-RUNTIME
k3s-server-1   Ready    control-plane,etcd,master   2m33s   v1.31.1+k3s-5e6dc725   172.17.0.4    <none>        K3s v1.31.1-k3s-5e6dc725   6.8.0-1016-aws   containerd://1.7.22-k3s1
k3s-server-2   Ready    etcd                        80s     v1.31.1+k3s-5e6dc725   172.17.0.5    <none>        K3s v1.31.1-k3s-5e6dc725   6.8.0-1016-aws   containerd://1.7.22-k3s1
k3s-server-3   Ready    control-plane,etcd,master   58s     v1.31.1+k3s-5e6dc725   172.17.0.6    <none>        K3s v1.31.1-k3s-5e6dc725   6.8.0-1016-aws   containerd://1.7.22-k3s1
k3s-server-4   Ready    control-plane,master        33s     v1.31.1+k3s-5e6dc725   172.17.0.7    <none>        K3s v1.31.1-k3s-5e6dc725   6.8.0-1016-aws   containerd://1.7.22-k3s1

Confirmed no breakage on the RKE2 side with these changes: rancher/rke2#7139

And the startup actually seems to be a bit faster, with less log spam while etcd is starting.