How to best pin an arbitrary part of a DAG ipfs/kubo#5142
Huge IPFS DAGs are likely to not fit in a single IPFS node. Additional datastore constraints (it is difficult to keep a non-sharded database of trillions of keys) may impose soft and hard limits on the amount of content that fits a single IPFS node. So...
How can we shard data in IPFS?
IPFS Cluster has code to split DAGs as they are added and create shards by generating an alternative DAG. We will explain how this works, the requirements and reasons why it is blocked waiting on some functionality from the ipfs daemon Pinner. Can we come up with better ways? Can we find a path to move forward? Sure we do!
We should be able to shard arbitrarily large DAGs.
https://cluster.ipfs.io/documentation/developer/rfcs/dag-sharding-rfc/
(warning, might be a little bit old and innacurate, but gives a good overview!)
🎤 Slides
TODO