Support gRPC's Server-side Connection Management

[cherrot]

Is your feature request related to a problem? Please describe.

https://github.com/grpc/proposal/blob/master/A9-server-side-conn-mgt.md

I will not elaborate on the benefits of implementing server-side connection management (you can refer to the original proposal above), but I need to explain why we urgently need this feature: communication within K8s clusters via service endpoint is by default implemented as a L4 LB, which prevents gRPC clients from promptly detecting service Pod scaling, leading to unbalanced loads. (In my case I heavily use bidi stream RPCs so HTTP/2 connection is inevitable)

However, if server-side connection management is supported, it can achieve load balance by actively disconnecting and reconnecting to the underlying gRPC service Pods.

Here is a good article (and illustration) explaining this resolution on gRPC world:

How three lines of configuration solved our gRPC scaling issues in Kubernetes

Describe the solution you'd like

Implement Server-side Connection Management. Specifically, the MAX_CONNECTION_AGE and MAX_CONNECTION_AGE_GRACE parts of the original proposal.

Additional context

I saw a similar feature request in connect-es project: connectrpc/connect-es#752

[jhump]

@cherrot, Connect implementations (not just this one, but others, too) generally rely on standard libraries for the transport layer. To that end, a Connect client or server does not know anything about the underlying connections/sockets. In the case of this repo, for Go, all that is handled by net/http. So I think you're looking for a custom net.Listener or perhaps HTTP middleware that could implement the above.

Having said that, you might be interested in a client-side solution: https://github.com/bufbuild/httplb.
That package can be configured for clients to automatically recycle connections after they reach a particular age (instead of the server doing it) using the WithRoundTripperMaxLifetime option. Also appropriate when using an L4 balancer is the resolver.WithMinConnections function, which can cause a client to create multiple outbound connections, even if the target name resolves to only a single IP address.

Also consider using a headless service in Kubernetes. Combined with the "httplb" library mentioned above, the client would resolve all of the backend pods (since DNS queries for headless services provide all pod IPs) and then perform client-side load balancing across them.

[cherrot]

@jhump Thank you for the detailed explanation!

From the original proposal:

MAX_CONNECTION_AGE is a duration for the maximum amount of time a connection may exist before it will be shut down. When a connection exceeds the configured age the server should send GOAWAY with status code NO_ERROR and ASCII debug data max_age and then continue the graceful connection termination.

I disagree with your perspective that server-side connection management is a matter of the transport layer. IMHO it is an application-layer logic based on the HTTP/2 GOAWAY frame implementation. Consider Connect supports multiple HTTP protocols (HTTP/1.1 and HTTP/2), it might be challenging to design a connection control mechanism that only works for HTTP/2 on top of this compatibility.

However, I think httplb could be an elegant replacement for server-side connection management! I hope it can be added to the official documentation as a best practice for deploying server-to-server RPC services under K8s 😄