Data redundancy demo?

[Winterhuman]

Thinking in terms of existing technology like RAID, or perhaps a more accurate comparison like Tahoe-LAFS, are there any plans for demonstrating redundant data replication across Iroh nodes, taking advantage of similar systems that the documents feature uses (mainly the PubSub stuff)?

[b5]

kinda. Iroh anchors will be backing up content they discover, but that's not really what you're talking about. The main problem replication management introduces is an additional consensus layer. Maybe we just use another document to manage replication?

Lots of additional inspiration to draw from here in addition to Tahoe-LAFS:

• Lockss
• [IPFS Cluster] (https://ipfscluster.io/) (cc @lanzafame)
• estuary

Do you have a need for this @Winterhuman?

[Winterhuman]

Less a need, more a curiosity if you've already been thinking about this. Package management comes to mind when thinking about possible use-cases, although this topic also has a lot of overlap with #839.

[Winterhuman]

Though as a point of interest, I've never read about a redundancy protocol that uses Blake3 before, and I have a suspicion that verifiable streaming could lead to some very strange (and possibly worth it's own thesis) optimisations. That's probably a long way aways, but I thought I'd mention it

[b5]

Yeah, @rklaehn already has iroh in a very sophisticated place in terms of storing & retrieving parts of blobs, he wrote #839 for a reason, as the use case is close to his interests. If you have time or interest in writing more detailed thoughts on the matter, I'd welcome them here, but for the moment I'll file this as "academic interests" until we can find a user with a burning need for the LOCKSS / RAID redundancy pattern

[Winterhuman]

Noteworthy projects, protocols, and papers (will update this comment over time):

Alpha Entanglement Codes

(Link to paper)

An "essentially" XOR-based error correction algorithm said to be even more fault tolerant than the Reed-Soloman algorithm, with a good range of customisibility in terms of the performance-disk_space-fault_tolerance trade-off. The first mention of this algorithm I had found came from Wondertan/go-ipfs-recovery#4, but https://github.com/dedis/student_22_ipfs_alpha_entanglement_code seems more relevant given its connection to IPFS-Cluster (and it also mentions another project using the algorithm; Snarl).