Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove DHT and bootstrapping from browser defaults #420

Open
aschmahmann opened this issue Feb 6, 2024 · 5 comments
Open

Remove DHT and bootstrapping from browser defaults #420

aschmahmann opened this issue Feb 6, 2024 · 5 comments
Labels
need/maintainers-input Needs input from the current maintainer(s) status/blocked Unable to be worked further until needs are met

Comments

@aschmahmann
Copy link

Background

Using the Amino DHT from browsers is currently problematic and expensive:

  • <1/3 of the Amino DHT supports transports accessible from a browser https://probelab.io/ipfsdht/#dht-transport-distribution
  • Almost all of those are WebTransport, which is currently only supported in Chromium (although Firefox support is close)
  • However, Chromium has artificial throttling of WebTransport connections which makes using them en-mass quite painful, particularly in p2p networks like the Amino DHT where peers may come and go since the throttling over-punishes dials to absent peers. https://issues.chromium.org/issues/40069954
  • The net result is that we currently spin up many connections in the browser that are themselves not useful which is a tremendous waste of resources and harms the overall experience

Because Helia is used for a variety of tasks spinning up expensive network calls that are unlikely to be useful is a problematic default. For instance, this helia example for working with a CAR spins up over 200 connections despite not needing the network at all. https://github.com/ipfs-examples/helia-examples/blob/c900274c2819ae320262009974d9a770e5a72955/examples/helia-create-car/src/provider/HeliaProvider.jsx#L38. While perhaps the example could change, I think it's worth considering that it's likely others will create examples in a similar way and it'd be better to give them a good onboarding experience by default.

Proposal

Let's remove the DHT and bootstrap-based peer discovery from the browser defaults. Such as:

dht: kadDHT({
clientMode: true,
validators: {
ipns: ipnsValidator
},
selectors: {
ipns: ipnsSelector
}
}),

peerDiscovery: [
bootstrap(bootstrapConfig)
],

What's Next?

As things like WebRTC-direct are rolled out by default in kubo and become more present in the Amino DHT and if/when the Chromium bugs are fixed it would be reasonable to evaluate re-enabling the Amino DHT by default, but IMO we should keep the defaults at what works best now rather than what we hope will be best later.

@achingbrain
Copy link
Member

The WebTransport throttling issue manifests itself as the WebTransport session erroring immediately after it's created. This means we try to dial a peer and fail, moving on to the next address to try again, causing a lot of thrashing in the application without yielding any viable connections to peers.

The primary use of the DHT is finding bitswap providers to service requests for CIDs. The ipfs-bitswap module does this by using libp2p content routing directly, of which @libp2p/kad-dht is an implementation.

#409 replaces ipfs-bitswap with @helia/bitswap - a lighter weight reimplementation - this uses the higher level Helia routing rather than dropping down to libp2p, so it can take advantage of @helia/delegated-routing-v1-http-api-client and hit HTTP delegate routers to find providers without requiring @libp2p/kad-dht or even libp2p at all.

The hope here is that we don't waste precious WebTransport connections dialling DHT peers, we can save them for bitswap operations. While this will be an improvement it is kicking the can down the road a bit, as at some point we'll hit the throttle limits and WebTransport will stop working until a page reload.

When WebRTC lands in go-libp2p and is turned on in Kubo by default this should get a lot better as it's a much better fit for distributed applications, given it's extensive use in peer to peer video conferencing.

So when we merge #409 we can remove the DHT from the browser build by default since we can use HTTP delegates to find providers, but we'll still run into problems actually fetching content via bitswap until WebRTC is usable.

In the interim, HTTP block brokers will be able to supply blocks for CIDs, though with some increased centralisation until they look up providers that support the IPFS HTTP Gateway protocol from HTTP delegates instead of hitting the same preconfigured trustless gateways over and over again.

@aschmahmann
Copy link
Author

I'm pretty sure we have a version of routing-v1 implemented in javascript that implements the libp2p routing interface and so is usable with ipfs-bitswap. Also, the defaults as-is are almost entirely harmful rather than helpful so removing them now (even if it would theoretically break ipfs-bitswap if the Amino DHT was usable from the browser) seems like an improvment.

That being said, if this is going to happen in a PR landing soon anyway (not sure the timeline on #409) then no use debating whether this should happen beforehand/in parallel.

@achingbrain
Copy link
Member

@helia/delegated-routing-v1-http-api-client supports the libp2p routing interfaces too, but it doesn't include the protocols supported by the providers, so it increases the chances of wasted dials, either trying to run identify or just blindly trying the protocol - I guess because it's largely modelled on KAD-DHT and the GET_PROVIDERS RPC call never had this information.

@2color
Copy link
Member

2color commented Feb 28, 2024

So when we merge #409 we can remove the DHT from the browser build by default since we can use HTTP delegates to find providers, but we'll still run into problems actually fetching content via bitswap until WebRTC is usable.

Why do we need to wait for that PR?

With the DHT Client enabled in browser by default, we're just setting up users for failure, given how low the chances are of establishing a connection.

@SgtPooki
Copy link
Member

SgtPooki commented Aug 8, 2024

go-libp2p has released version with webrtc, so this should be released in Kubo soon, and then we can move forward with these changes

@SgtPooki SgtPooki added the need/maintainers-input Needs input from the current maintainer(s) label Aug 8, 2024
@SgtPooki SgtPooki added the status/blocked Unable to be worked further until needs are met label Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
need/maintainers-input Needs input from the current maintainer(s) status/blocked Unable to be worked further until needs are met
Projects
None yet
Development

No branches or pull requests

4 participants