Skip to content
This repository has been archived by the owner on Sep 21, 2024. It is now read-only.

feat!: Traverse the Noosphere vast #284

Merged
merged 3 commits into from
Mar 23, 2023
Merged

feat!: Traverse the Noosphere vast #284

merged 3 commits into from
Mar 23, 2023

Conversation

cdata
Copy link
Collaborator

@cdata cdata commented Mar 23, 2023

Building on the work in #278 and #283 , this change implements our first take on sphere traversal as outlined in #269. By invoking SphereContext::traverse_by_petname, a user can move from one sphere to the next with minimum information about the sphere's data format or its relative location on the network.

This change implements sphere traversal in Rust, but stops short of adding an FFI; I'll add the FFI in a smaller follow-up change.

Important note: this change breaks our HAMT's data layout. In the past, HAMT values were stored and encoded directly inside a Vec<KeyValuePair>. As of this change, KeyValuePair internally holds a link to the data, not the data itself. This reduces the amount of data we have to store, but increases the number of lookups we have to do from storage in order to read the values of a HAMT. We are compensated for this trade-off by transferring less over the wire during replication (and it also simplifies the logic of determining what to stream when replicating).

Fixes #269
Fixes #2

@cdata cdata requested a review from jsantell March 23, 2023 01:23
jsantell
jsantell previously approved these changes Mar 23, 2023
Copy link
Contributor

@jsantell jsantell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Can't wait to see this in action

rust/noosphere-collections/src/hamt/test.rs Show resolved Hide resolved
rust/noosphere-gateway/src/worker/mod.rs Outdated Show resolved Hide resolved
rust/noosphere-gateway/src/route/replicate.rs Show resolved Hide resolved
rust/noosphere-gateway/src/route/replicate.rs Show resolved Hide resolved
rust/noosphere-gateway/src/worker/syndication.rs Outdated Show resolved Hide resolved
rust/noosphere-storage/src/retry.rs Show resolved Hide resolved
rust/noosphere-storage/src/tap.rs Show resolved Hide resolved
rust/noosphere/src/noosphere.rs Outdated Show resolved Hide resolved
rust/noosphere-sphere/src/replication.rs Show resolved Hide resolved
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable traversal from one sphere to the next Change Bundle to enable a streaming interface
2 participants