Petname resolution and synchronization in spheres and gateways #253

cdata · 2023-03-07T20:26:41Z

This change pertains to #12 and started as an effort to close the loop on end-to-end name publishing and resolution. Unfortunately, one thing lead to another and the change came to embody a significant refactor of how we read from and write to spheres in general.

Publishing and resolving names

As of this change, we have implemented everything needed for a user to publish a version of their sphere to the Noosphere Name System, and for another user in the distance to resolve that published version by petname. We've also stubbed out the corresponding C FFI in preparation for #243.

Petname entries are added by a user to their own sphere. When the user sync's with their gateway, the gateway adopts the user's petname changes and kicks off a background job to resolve all the updated names from the Noosphere Name System. As those names are resolved, the gateway incorporates the resolved values into its address book. The next time a user sync's with the gateway after the resolutions are complete, the resolved values propagate back to their local device, and the sync strategy incorporates the new resolutions into the user's own address book.

Publishing happens on every push of the user's history to the sphere. It is technically optional to publish; it requires a specially crafted UCAN invocation, which is marked as optional in the body of the API request. But, we currently do it on every push anyway. When the user pushes, the gateway forwards the updated record to the Noosphere Name System in a background thread.

The `SphereContext` refactor

The rationale for the refactor is that previously we had only one domain where we were performing a lot of user-driven reads and writes: SphereFs. We had factored things such that SphereFs did internal accounting of pending mutation state to a sphere. This immediately became a problem when we moved to add reading and writing APIs for a different domain (petnames), because SphereFs only dealt with the content part of a sphere.

The new factoring discards SphereFs and replaces it with a series of traits that ship the same API, but that are implemented directly for containers such as Arc<SphereContext<_, _>>. So, where previously we used one construct as a handle to a sphere - SphereContext - and a different one as a read/write API - SphereFs - we now do everything with SphereContext. New traits have been introduced to deal with petnames in a similar style to the traits that deal with content. And, all pending mutations are now "centrally" tracked by the SphereContext itself, so there is no risk of forking history by writing to two different parts of the sphere at the same time.

Fixes #246
Fixes #214
Fixes #215

jsantell

Looking great! some early comments so far

rust/noosphere-core/src/view/sphere.rs

jsantell · 2023-03-07T23:46:59Z

rust/noosphere-core/src/view/sphere.rs

+
+    /// Consume the [Sphere] and get a [Stream] that yields the [ChangelogIpld]
+    /// for content slugs at each version of the sphere.
+    pub fn into_link_changelog_stream(


Depending on future plans for these stream functions, they could maybe have an enum or query interface, e.g.

enum SphereMetaType { Name, Link }; into_changelog_stream(self, SphereMetaType, Option<&Cid>)

or

struct ChangeLogQuery { since: Option<&Cid>, type: SphereMetaType, } into_changelog_stream(self, ChangeLogQuery);

We might have to manifest this as a macro helper. Changes here refer to the change in a VersionedMap and are typed Changelog<K, V>, so the method invocation needs to imply key and value types in addition to the field.

rust/noosphere-core/src/view/versioned_map.rs

rust/noosphere-gateway/src/nns.rs

rust/noosphere-sphere/src/sync/gateway.rs

jsantell · 2023-03-08T16:30:55Z

rust/noosphere/src/ffi/petname.rs

+#[ffi_export]
+/// Assign a DID to a petname. This will overwrite a petname entry if one already exists
+/// with the given name (and reset the resolved CID, if any).
+pub fn ns_sphere_petname_set(


The tuple-nature of the address book ((id, petname) => cid) doesn't lead to obvious FFI patterns. We should document outcome if petname/did are NULL (implementation of course TBD). I don't think I'd expect the petname to be the "key" here, but changing DID or petname are both valid use cases. Wonder what it'd look like if e.g.

links = ns_sphere_links_get(); for (uint8_t i = 0; i < links.len; i++) { ns_link* link = links.array[i]; printf("name: %s", ns_link_get_name(link)); printf("did: %s", ns_link_get_did(link)); ns_link_set_name(link, "alice"); }

Though may cause similar synchronization issues as separation of Fs from Context.

Note that under the hood, names (like slugs) are keys in a HAMT. This data structure is convenient (and necessary) because it allows us operate on the data with map semantics, while also allowing us to avoid loading in the entire data structure just to read one key.

The implication here is that loading in the entire set of names could be quite expensive as a general purpose operation, especially if we are working on one key at a time. It's expensive in terms of memory allocation and it's expensive in terms of storage access because the key space of the "map" is effectively sharded across many blocks.

That having been said, even in cases where synchronization is not an issue, we would still want to mediate access to the list of petnames via FFI. And, on the Rust side we implement iteration over all petnames/slugs/etc. as Streams - which are effectively async generators - so that we can side-step the pathological case of loading the entire key space into memory at once.

So, I guess the takeaway here is that we need a FFI to access the full space of petnames.

Well explained, thanks!

rust/noosphere/src/ffi/petname.rs

rust/noosphere-api/Cargo.toml

jsantell

Handful of comments with a closer eye on the NS changes, and some small things like comments. Excellent, gigantic work!!

jsantell · 2023-03-09T22:53:28Z

rust/noosphere-cli/src/native/commands/sphere.rs

@@ -101,6 +98,7 @@ Type or paste the code here and press enter:"#,
    let cid = Cid::from_str(cid_string.trim())
        .map_err(|_| anyhow!("Could not parse the authorization identity as a CID"))?;

+    debug!("A");


nit: remove or clarify

jsantell · 2023-03-09T22:53:32Z

rust/noosphere-cli/src/native/commands/sphere.rs

@@ -109,6 +107,7 @@ Type or paste the code here and press enter:"#,
        .authorized_by(Some(&Authorization::Cid(cid)))
        .build()
        .await?;
+    debug!("B");


nit: remove or clarify

Explanation: b

jsantell · 2023-03-09T23:10:17Z

rust/noosphere-sphere/src/sync/gateway.rs

+        if updated_names.len() == 0 {
+            return Ok(None);
+        }
+        println!(


No println in noosphere imo, this should be rendered in orb if needed.

There's plenty of existing println here though, let's kick off our rules to writing to stdout in a separate issue

jsantell · 2023-03-09T23:10:35Z

rust/noosphere-sphere/src/sync/gateway.rs

+            if let Some(jwt) = last_known_record {
+                context.adopt_petname(&name, &identity, &jwt).await?;
+            } else {
+                warn!("UH OH");


jsantell · 2023-03-09T23:15:13Z

rust/noosphere-sphere/src/sync/gateway.rs

+                },
+                can: SphereAction::Publish,
+            })
+            .with_lifetime(120)


Wouldn't this record expire in 2 minutes? We should have some constant, default lifetime duration somewhere

SG, although worth noting that at this time we have no expiration of name records (at least, not one that we enforce).

Already filed #259 to discuss

jsantell · 2023-03-09T23:32:49Z

rust/noosphere-ns/src/dht/processor.rs

+        config: DhtConfig,
+        processor: DhtMessageProcessor,
+    ) -> Result<tokio::task::JoinHandle<Result<(), DhtError>>, DhtError> {
+        // debug!("SPAWNING WITH VALIDATOR: {:?}", validator);


nit: comment

jsantell · 2023-03-09T23:40:42Z

rust/noosphere-ns/src/name_system.rs

-        store: SphereDb<S>,
-        dht_config: DHTConfig,
+        dht_config: DhtConfig,
+        validator: Option<V>,


I don't like a configurable/optional top-level validator (wanting a network of nodes to have relatively similar/consistent validators), but I don't have any other ideas in order to support this (and more pushing and pulling between static DhtConfig values and living instances. I'm glad you've spent some time in this module and the Dht* counterparts for future discussions on some of the cleanup challenges ahead!

Pasting my related comment from Discord:

I'm sympathetic to your position on configurable Validator. Ultimately, I don't think orb-ns will offer a configuration here. Nor would orb. And, it may prove to be an anti-pattern altogether.

And filed #268

jsantell · 2023-03-09T23:42:05Z

rust/noosphere-ns/src/records.rs

@@ -12,7 +12,7 @@ use std::{convert::TryFrom, str, str::FromStr};
 use ucan::{builder::UcanBuilder, crypto::KeyMaterial};
 use ucan::{chain::ProofChain, crypto::did::DidParser, Ucan};

-/// An [NSRecord] is the internal representation of a mapping from a
+/// An [NsRecord] is the internal representation of a mapping from a


I'm not ready to talk about how NsRecord/NSRecord is the only struct prefixed by NS 🙃

jsantell · 2023-03-09T23:44:32Z

rust/noosphere-ns/src/helpers.rs

+        .build()?
+        .sign()
+        .await?;
+    let _ = store.write_token(&delegation.encode()?).await?;


Kind of a bummer that store is being passed around these utils only for this side-stepping-Ipfs hack here, now that store is internal to the validator

jsantell · 2023-03-09T23:45:27Z

rust/noosphere-ns/src/validator.rs

@@ -22,17 +22,31 @@ where
    }
 }

+impl<S> Clone for Validator<S>


Should this also implement Display like the AllowAllValidator?

Ah, Display is vestigial, I can remove that implementation

cdata · 2023-03-10T01:46:43Z

Alright folks, we're gonna land this one.

…#253)

cdata force-pushed the feat/resolve-names-in-gateway branch 2 times, most recently from 50ff040 to ae6997c Compare March 7, 2023 20:39

jsantell reviewed Mar 8, 2023

View reviewed changes

cdata force-pushed the feat/resolve-names-in-gateway branch from ae6997c to 4add4d0 Compare March 8, 2023 19:27

jsantell mentioned this pull request Mar 8, 2023

Reconfigure rust module dependencies #254

Merged

cdata force-pushed the feat/resolve-names-in-gateway branch from ff57c8e to 772139c Compare March 9, 2023 01:33

cdata marked this pull request as ready for review March 9, 2023 02:02

cdata requested a review from bfollington March 9, 2023 02:03

cdata added 5 commits March 9, 2023 10:43

feat!: SphereContext refactor

96706e9

chore: Add E2E integration test for petnames

698cc84

chore: Annotate TODOs with issue numbers

8cd8d98

docs: Add missing comments, clean up a typo

1b43b7f

chore!: Remove try_ prefixes from many APIs

e3e3145

cdata force-pushed the feat/resolve-names-in-gateway branch 4 times, most recently from 5b15264 to 04e28fc Compare March 9, 2023 19:48

chore: Fix container image workflow concurrency

d74e0c0

cdata force-pushed the feat/resolve-names-in-gateway branch from 04e28fc to d74e0c0 Compare March 9, 2023 19:52

jsantell previously approved these changes Mar 9, 2023

View reviewed changes

cdata mentioned this pull request Mar 10, 2023

Remove configurable validation from NNS #268

Closed

chore: Cleanup per PR feedback

a2a209b

cdata dismissed jsantell’s stale review via a2a209b March 10, 2023 01:21

cdata merged commit 95ea405 into main Mar 10, 2023

cdata added a commit that referenced this pull request Mar 10, 2023

feat!: Petname resolution and synchronization in spheres and gateways (…

f7ddfa7

…#253)

github-actions bot mentioned this pull request Mar 10, 2023

chore: release main #236

Merged

This was referenced Mar 10, 2023

dev: Integration of NameSystem within the CLI v1 #191

Closed

Integrate Noosphere Name System with CLI gateway #109

Closed

cdata mentioned this pull request Mar 13, 2023

feat!: Implement C FFI for petname management #271

Merged

cdata mentioned this pull request Mar 26, 2023

feat!: Sphere traversal C FFI #292

Merged

cdata deleted the feat/resolve-names-in-gateway branch April 18, 2023 00:47

This was referenced May 8, 2023

chore: release main #364

Closed

chore: release main #367

Closed

chore: release main #371

Closed

This was referenced Jul 1, 2023

chore: release main #457

Closed

chore: release main #458

Merged

cdata mentioned this pull request Aug 28, 2023

LinkRecord can propagate before blocks are available #606

Open

github-actions bot mentioned this pull request Aug 29, 2023

chore: release main #616

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Petname resolution and synchronization in spheres and gateways #253

Petname resolution and synchronization in spheres and gateways #253

cdata commented Mar 7, 2023 •

edited

Loading

jsantell left a comment

jsantell Mar 7, 2023

cdata Mar 9, 2023 •

edited

Loading

jsantell Mar 8, 2023

cdata Mar 9, 2023 •

edited

Loading

jsantell Mar 9, 2023

jsantell left a comment

jsantell Mar 9, 2023

jsantell Mar 9, 2023

cdata Mar 10, 2023

jsantell Mar 9, 2023

jsantell Mar 9, 2023

jsantell Mar 9, 2023

cdata Mar 10, 2023

jsantell Mar 9, 2023

jsantell Mar 9, 2023

cdata Mar 10, 2023

jsantell Mar 9, 2023

cdata Mar 10, 2023

jsantell Mar 9, 2023

jsantell Mar 9, 2023

cdata Mar 9, 2023

cdata commented Mar 10, 2023

Petname resolution and synchronization in spheres and gateways #253

Petname resolution and synchronization in spheres and gateways #253

Conversation

cdata commented Mar 7, 2023 • edited Loading

Publishing and resolving names

The SphereContext refactor

jsantell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdata Mar 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdata Mar 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsantell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdata commented Mar 10, 2023

cdata commented Mar 7, 2023 •

edited

Loading

The `SphereContext` refactor

cdata Mar 9, 2023 •

edited

Loading

cdata Mar 9, 2023 •

edited

Loading