Skip to content

Latest commit

 

History

History
90 lines (53 loc) · 3 KB

sharding.md

File metadata and controls

90 lines (53 loc) · 3 KB

Sharding

Some background

Sharding in Peerbit is based on the content being committed. Each commit will be added to a log that represents some meaningful state, like an image or a document.

Every change in Peerbit are committed with explicit links to the content it depends on. This means that by following the dependencies to the root, we can get the full state

p1

Every graph gets an graph id (GID)

p1

When graphs merge (two independent states becomes dependent) the new graph will be named the same as the graph with the longest chain

p1

p1

Now this is important to background in order to understand how replicators/content leaders are chosen based on new changes.

The distribution algorithm

Imagine the commit above is made, so that the merged graph gets the label "DOG", how can we choose replicators in a fully connected network in a simple random way? (By being a replicator you have the task of storing the log and potentially also make it searchable for peers)

p1

1.

The first thing we need to do is to hash the labels of the peers (IPFS IDs) and the DOG label with a hash function (more details on this function later)

p2

2.

Secondly put all the hashes into a list and sort it.

p3

3.

Now we lookup the labels from the hashes again

p4

4.

Now if we want 2 replicas of our content we can choose that the replicators are the 2 next elements in the list

p5

The hash function is seeded with the checksum of the content itself, so it changes for every new commit. This means that the results would differ if the content changes. E.g.

p5

When graphs merges and peers joins and leaves

When peers leave and join we need to redo leader selection for the heads of our content, this is because there might be replicators that no longer are online, or there might be new peers that should be replicators instead of someone else since the outcome of the algorithm is dependent on what peers participate in the replication process.

Similarly graphs merge (like when the CAT and DOG became a DOG), this is functionally equivalent to that replicators of CAT stop to replicate and that DOG replicators start to replicate a larger log.

Implementation

The implementation can be found here (findLeaders method)