[EPIC] BlockSync Overhaul - Storage

[Wondertan]

Context

This EPIC is an entry point for the BlockSync Storage Overhaul efforts.
For deeper context, look at ADR document: #1037

Tasks

Preliminary Work

• ipld: Consolidation with share #878
• fraud: Move BEFP definitions to share pkg #1100
• deps: bump rsmt2d to latest #824
• IPLD/NMT improvements #183
• share/ipld: Remove duplicated namespace appended by NMTWrapper #1239
• Fork DAGStore and extended ShardAccessor to return an io.Reader #1101
• share: Extract thread-safe EDS definition out of Retriever.retrieverSession #1103 (Nice to have)

EDS Serde

• share: WriteEDS #1105
• share: ReadEDS #1104
• feat(share/eds): ODSReader #1352
• share: benchmark Read/WriteEDS #1106

EDSStore

• share: EDSStore definition and constructor #1107
• share: EDSStore.Put #1108
• share: EDSStore.GetCAR #1109
• share: EDSStore.Blockstore #1115
• share: EDSStore.Remove #1110
• share: EDSStore.Get #1111
• share: EDSStore.Has #1112
• share: Periodic GC over DAGStore inside EDSStore #1116
• node: Integrate share.EDSStore #1114
• node: Ensure EDStore has EDS of an empty block #1113
• share: metrics and traces for EDSStore #1361
• eds: Store.CARBlockstore #1427
• eds: Use DataHash instead of share.Root in Store methods #1428
• share/eds: Store.GetRoot #1509

Reading DataByNamespace

• share: Extend GetSharesByNamespace with an option to collect Merkle proofs #1117
• share: GetCARByNamespace #1118

Bridge/Full Node Integration

• header/core: move Listener and Exchange out to the root core pkg #1424
• node: migrate Blockstore over eds.Store.Blockstore for FN/BNs #1422
• header/core: Put Blocks into the eds.Store in Listener #1423

Post Housekeeping/Optimizations

• Mergeshare.CacheAvailability and share.LightAvailability #1119
• das: Revisit DASer concurrency default params for the Full Node #2743
• share: Restructure into sub pkgs #1120
• share: write doc.go to be Go Doc compliant
• node: revisit the need for the Blockstore caching with bloomfilter #2742
• share: revisit APIs and minimize public surface in the pkg and subpkgs #2744
• share/eds: WriteEDS: don't store parity and namespaces copies #1233
• share: consistent usage of sha256-minio #1291
• share/eds: Store should cache EDSes on Put #2018
• eds: Blockstore provided by the Store should support Put #2424

Related

• Revamp allocations on rsmt2d #1121
• Research Bitswap extension to avoid the top-level index in share.EDStore #1122

BugFixes

An ongoing list of bugfixes that are discovered and fixed during the EPIC implementation:

• fix(ipld/plugin): don't truncate a type byte when it's not in the data #1196
• fix(eds): WriteEDS thread safety for concurrent writingSessions #1498
• eds/store: Fix ShardAccessor caching for use in GetCAR and CARBlockstore #1514