Sharing QUIC connection between protocols / support protocol extensions

[Frando]

I wanted to open a discussion on reusing Iroh connections for application-level protocols. I'll outline some usecases, and to get a discussion started I'll also add some very preliminary thoughts on a potential implementation that came to mind when reading the Iroh source code and thinking about existing multistream implementations (libp2p, hypercore protocol).

background and usecases

At the moment Iroh owns a QUIC connection, as in: The provide server treats each stream as an Iroh protocol stream. Especially with the holepunching features getting in, it will be very attractive to reuse the Iroh connection establishment setup, and run both the Iroh protocol and other application-specific protocols over the same quic connection.

Use cases are wide:

• In Deltachat, a future direction is to support application level peer streams for WebXDCs. Reusing the Iroh QUIC-TLS handshaking and holepunching (once landed) would be great.
• For other applications integrating Iroh for blob transfer, there will certainly be needs to send other data of some form that doesn't easily fit the Iroh protocol (e.g. other query formats than the Iroh set/collection query, where the query resolves to a list of hashes that are then to be synced with the Iroh protocol). Again, it would be great to reuse the Iroh holepunching / handshaking / ticketing infrastructure instead of having to use two different connection stacks between peers.
• It would also simplify experimenting with novel data transfer protocols or other extensions

ideas for implementation

In libp2p land, this is solved by always sending a multistream protocol identifier on each substream first.

To reduce the overhead of sending a protocol string identifier for each request (if there are to be many), this could be optimized quite easily by assigning an integer ID to the protocol on first use. So the string identifier is only sent once per connection, subsequent streams use the int id. Hypercore Protocol uses the same technique for protocol extensions.

The current Iroh protocol currently requires a Handshake message as first message on each new stream. Which likely is to be optimized anyways once Iroh starts to use many requests / streams to not resend the 32 byte auth token on each request. So possibly this could be combined quite nicely:

enum Handshake {
  RegisterProtocol { id: u16, protocol_name: String, handshake: Bytes  },
  ReuseProtocol(u16)
}

The current Iroh handshake with protocol version and auth token could live in the opaque handshake of such a new multi-protocol handshake. Subsequent requests would use the id that the client assigned on the first request and skip the protocol name and handshake bytes. For non-Iroh protocols, the raw quinn (SendStream, RecvStream) could be passed on to the application after decoding the handshake message.

For the library use case, exposing the streams that are not handled in Iroh I could see working in two ways:

• either iroh passes a connection back to the application to loop over incoming streams, and the application then passes Iroh streams back to Iroh
• or the iroh server could expose a method like register_protocol(name: &str, on_stream: impl Fn(stream: (quinn::SendStream, quinn::RecvStream), handshake: Option<Bytes>)

In the binary, unknown protocols could just be ignored.

[flub]

I don't think we've quite agreed that iroh wants to provide this functionality, but personally I find it rather tempting too. However I think we're going to concentrate on getting the base protocol reliable and fast before we start adding other things. With those caveats out of the way, here are my current thoughts:

• As already mentioned the iroh protocol itself will want to use multiple streams. I'm not really sure yet I'd like to send an identifying frame on each stream. But maybe. Dynamic registration doesn't make me very enthusiastic though, that seems like a really hard problem to get right.

• Not only are multiple streams cheap in QUIC, also multiple connections between endpoints are supposed to be relatively cheap. E.g. https://www.rfc-editor.org/rfc/rfc9308.html#name-stream-versus-flow-multiple says:

  Given QUIC's ability to send application data in the first RTT of a connection (if a previous connection to the same host has been successfully established to provide the necessary credentials), the cost of establishing another connection is extremely low.

  So I would be very tempted to explore multiple QUIC connections to expose the transport to users.

• One mayor downside of doing this is that, as the same RFC section points out, you get a different flow, so you can no longer pack frames from multiple connections in the same UDP datagram. Which would be a loss for sending small things. However, it is not impossible that the underlying "magic socket" from the holepunching could still do that. We'll have to see.

[rklaehn]

So I like the idea of using separate connections via the same magic socket. That way we would not have to walk on eggshells in our own protocol evolution, but would still share the hole punching mechanism.

Regarding identifiers, I am also not fond of some dynamic string - id mapping. Maybe just have a table somewhere in this repo where people can reserve an integer for their extension.

[Winterhuman]

Reserving integers/bytes for each protocol is very similar to how the multicodec table works (https://github.com/multiformats/multicodec). However, I'd like to mention that the approach can be a bit rigid when trying to work with experimental protocols, sometimes devs aren't ready to commit to an ID yet, but still want it to be negotiable (see multiformats/multicodec#191 or multiformats/multicodec#39).

Having a way to negotiate protocols without having an ID on the table should be considered to avoid this.