Iroh as an IPFS-compatible library

[emmyoh]

Hello!

To preface this discussion post, I want to say that I recognise and appreciate all the hard work that has gone into this project. Please, don't interpret any of what I say as either being criticism or a demand—Iroh, as it currently is, is brilliant in its own right.

I wanted to share that I've been following Iroh as someone who was hoping for a mature, functioning implementation of IPFS that I could embed in my projects. Before the beetle & iroh split, it seemed like, with iroh-embed, that this was going to be exactly that. I know that the current version of Iroh aims to be, essentially, IPFS 2.0—not a completely different thing, but an evolution that is not compatible with the IPFS reference implementation (kubo).

I may be alone in this, but being able to grab pre-existing content on the IPFS network is fairly important to whatever I would use Iroh for. The thought was, I could add Iroh as a dependency to my projects to enable them to grab whatever is currently on the IPFS network. I understand that the current plan is to eventually revisit beetle at some indefinite future date, but I think it'd be important to instead have an actively-maintained component of Iroh that aims to be compatible with the pre-existing IPFS network. The utility of IPFS isn't just the protocol—I actually want some way of accessing it programmatically with my Rust projects, and I imagine I'm not actually alone in having this desire. Right now, for as cool as Iroh is independently (and I recognise that it has so much potential to transform the IPFS community!), I can't actually use it as an embedded Rust IPFS implementation.

I know you don't have infinite resources—you must prioritise what you work on—but I'd say beetle (or, rather, an optional kubo compatibility layer for the Iroh library) is critical for getting more people to start using Iroh. That's just my take, and if you feel completely differently then that's OK. Regardless, best wishes and congrats on building such a fantastic & innovative product!

[fabricedesre]

Hi @emmyoh,

I'm not a core iroh/beetle dev, but I currently use a fork of beetle (https://github.com/capyloon/beetle/tree/capyloon). I would say that depending on your use case, you want to keep using Beetle or not:

• if you just need "read" access, ie. fetching ipfs content, there's a least one other implementation that started and could be a better fit as a lightweight embedded solution - I'll post a link to the repo once it starts to be usable.
• if you need to provide content to the network, I think beetle is still your only option.

[emmyoh]

Thanks for your response! I'll check out both repositories that you mentioned!

[b5]

I may be alone in this, but being able to grab pre-existing content on the IPFS network is fairly important to whatever I would use Iroh for.

You are definitely not alone on this, and we need to do a better job of giving solutions for the numerous folks who need to work with both Iroh and Kubo. And I have to apologize in advance, the process of bridging from Iroh to kubo will likely take us longer than you'd hope. We are trying our best to keep iroh as a "do one thing well" piece of software. Interop with kubo introduces complexity we need to manage carefully to avoid sending you or any other user in the wrong direction.

With that said, we get it, the divide between Iroh & Kubo is a huge barrier that we need to address quickly. Our plan at the moment is to get going on some clear docs, utility crates & CLI tools that help close the gap.

At the same time, we're gathering use cases to understand what folks actually need, for software they intend to maintain long term, and build any necessary additions to iroh from there.

So If it's alright with you @emmyoh, I'd like to ask a bunch of questions about the specific things you'd like to do with kubo & Iroh, so we can build things that work. I'm sure my answers will be unsatisfying at first, but hopefully over time we can identify the right solutions to interop that balance performance with convenience. We deeply appreciate anyone & everyone's patience while we figure out the right way to do this.

My questions:

grabbing pre-existing content from the IPFS network

• What kind of content are you grabbing? is it mainly UnixFS stuff, IPLD, or a mix?
• Rough guess of byte size ranges?

I ask because we might be able to get away with a racing gateway shim for fetching content, which would issue HTTP requests to a configurable set of gateways, and verify the content on receipt.

An embeddable IPFS library

• What platforms are you targeting?
• What are your expectations of session length of the parent process?
• Would your application benefit from "dumb pipes" that let you do arbitrary networking to peers beyond data transfer?

Feel free to answer any, all, or none of these questions. We'll do our best to smooth over the gap between kubo & iroh, but more than that hopefully we can demonstrate that we're listening to what you want to do, and hope to help you get there

[emmyoh]

What kind of content are you grabbing? is it mainly UnixFS stuff, IPLD, or a mix?

I'm trying to build utilities which grab content using ipfs:// URIs with CIDs, which, I believe, would mean I'd be grabbing IPLD content and not UnixFS? My uncertainty comes from the fact that I'm more familiar with IPFS from the perspective of a tech enthusiast interested in the technology, and not as someone who understands the technical details of how it works (& the related jargon), and what all the technical components are.

Rough guess of byte size ranges?

The tools that I'm building which could benefit from IPFS support are concerned with grabbing both HTML documents and media files (images, videos, audio, etc)—as such, I can't really give a useful range, other than 'data on the same scale as what your browser commonly fetches on the Web'.

I ask because we might be able to get away with a racing gateway shim for fetching content, which would issue HTTP requests to a configurable set of gateways, and verify the content on receipt.

I may be misunderstanding what you're saying here—I apologise if I am; there is a pre-existing crate which uses Kubo's HTTP API to fetch content. I want to avoid the user having to run a separate daemon (such as Kubo serving its HTTP-accessible gateway), and I want to avoid making HTTP requests to publicly-accessible gateways.

What platforms are you targeting?

Personally, I'm only concerned with having support for Linux, macOS, and Windows, in that order. Mobile and WASM aren't concerns for me personally, but I could see many people wanting an IPFS implementation for the Web that isn't ipfs-js.

What are your expectations of session length of the parent process?

I have two Rust projects that I want having IPFS support as soon as possible:

1. An image viewer in the terminal which can fetch images using URIs
2. A Web browser which can support IPFS in addition to HTTP(S)

The session lengths here are quite different—the first has a session lasting under a second (a single command-line invocation of the program), and the second has a session time of, potentially, several continuous hours.

Would your application benefit from "dumb pipes" that let you do arbitrary networking to peers beyond data transfer?

My priorities are, personally, the following:

1. Grabbing IPFS content as easily as one would fetch HTTP content
2. Having the ability to share files with the rest of the IPFS network. A goal for my IPFS-powered Web browser project is that one would be able to share their local files & folders with a click of a button and get a shareable CID

Any peer-to-peer capabilities beyond this aren't relevant to me. I think it'd be quite nice to have some more applications for IPFS beyond P2P file sharing—something more like the dat/Hypercore protocol, which can support mutable P2P content, P2P live streams, etc—but I think that'd be beyond the scope & purpose of IPFS.

If any of my responses were unclear, ambiguous, or difficult to follow, I'm happy to clarify any of what I said. Wish I understood more of how this all worked—it's all very exciting stuff, I just tend to learn by using rather than reading, and so I figured I'd understand more about P2P protocols by trying to use an API with something I build.

[b5]

This is incredibly helpful, thanks so much for taking the time to write this, and all of your other contributions to the Iroh discussion @emmyoh!

I think I have a pretty clear sense of where to go from here. You've summed it up well with:

I just tend to learn by using rather than reading

I think the right next step is to get working on a tutorial that outlines grabbing IPFS content with a combination of Iroh & beetle libraries, then work toward a second example that details sharing content. This may take us some time to put together, but give me 2-3 weeks & I'll circle back with something 😄

[emmyoh]

Greatly appreciate how responsive you and the other Iroh contributors have been; I can tell that there is a uniquely talented & passionate team behind this project. Look forward to hearing from you then!

[emmyoh]

This is incredibly helpful, thanks so much for taking the time to write this, and all of your other contributions to the Iroh discussion @emmyoh!

I think I have a pretty clear sense of where to go from here. You've summed it up well with:

I just tend to learn by using rather than reading

I think the right next step is to get working on a tutorial that outlines grabbing IPFS content with a combination of Iroh & beetle libraries, then work toward a second example that details sharing content. This may take us some time to put together, but give me 2-3 weeks & I'll circle back with something 😄

Hi @b5 ,
Wanted to check in and see if you've made progress on those tutorials that you mentioned? If not, I completely understand—I appreciate the effort going into this project and you seem very busy. If you have the time, I'd appreciate a brief look at how to fetch content with Iroh & Beetle. Hope you've been well!