Vaults should have default nodes acting as remotes

[CMCDragonkai]

Specification

1. Doing pk vault pull some-vault should default to a particular node id
2. This should be the equivalent of git remotes, but we don't have to use the remote system in isogit for this, we could have this be tracked inside the vaultmanager system
3. Should evaluate if git remotes can be used for this though

Additional context

See the developer environment usecase

Tasks

1. - Change vaultIds from random bytes to a generated uuid with the nodeId appended (this will mean the nodeId will have to be passed into VM)
2. - When using the clone function, ensure that the new vault has the same vaultId as the vault it was cloned from (plus the appended nodeId)
3. - Change the pull function to look up the linked node in the node map (by default it should have the vault that it was cloned from
4. - Write a function that updates the default node Id for a given vault
5. - Write tests for default pulling and updating linked node id

Resolved in https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/205

[CMCDragonkai]

Note that use git branch -vv to see how branches are related to remotes.

Our vault have only 1 branch, that being of master.

They should then be able to pull from a default NodeId to reduce friction.

This default remote can be updated to point to another NodeId.

[scottmmorris]

This may need a reworking of the current GitManager class. I realise this may have been the better way to do this from the beginning.

• User passes in the vaultName of the local vault to pull updates to
• Call getVaultId to retreive the Id of the local vault to pull updates to
• Pass the vaultId of the local vault to pull updates to int GitManager pullVault function
• The vault map can then maintain details about the default remote locations to pull from
• GitManager looks up the linked nodeId and connects to that node
• GitManager looks up the linked vaultId

Here we run into a problems in that names can be changed. Ids though remain the same which means you could store the remote as a vaultId, meaning you would also need to store the nodeId. However If you wanted to change the default nodeId you would have to know the vaultId of the vault linked to the original vault of the original node., This becomes too complicated.

Here is the proposal:

• VaultMap maintains a VaultId -> VaultLink connection
• The VaultLink contains the nodeId of the default remote and the VaultId of the original cloned vault
• On initializing a vault, the nodeId is set as the nodes own nodeId and the vaultId is set as the vaultId of the vault in question
• When cloning a vault, the nodeID of the connected remote is stored in the vault map but the vaultId is obtained from the vault that it connected to
• When pulling, the nodeId and vaultId is looked up to connect and pull the write information

[scottmmorris]

Although should you even be able to pull changes from a vault that is not the original vault? Does iso-git even allow this? I will test now.

[CMCDragonkai]

Yes from the point of view of git. We change git remotes all the time. However from the POV of Polykey this should only be allowed for vaults with the same vault id. Which should be preserved when shared between Keynodes.

…

On 7/13/21 3:56 PM, scottmmorris wrote: Although should you even be able to pull changes from a vault that is not the original vault? Does iso-git even allow this? I will test now. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#208 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE4OHJ2ZM2JDVAHL4RXHGTTXPISHANCNFSM5ACHGBNQ>.

[scottmmorris]

The vault id of a cloned vault is currently not preserved between nodes, if it was then this might cause conflicts. For example if a node clones the same vault twice, they would end up with an error as they own two vaults with the same vault id.

As in my comment above wouldn't it be better to maintain a copy of the original vault id in the vault map which can be referred to when cloning?

[CMCDragonkai]

I thought the vault IDs were made unique so that they can be maintained across keynodes. This ensures that knows that it's the same vault even if the vault names can be different. Ensuring that all vault IDs are different across keynodes means you have to do bookkeeping on which vault id you are tracking. But is this necessary? We can make vaultids never conflict by making use of the NodeId. That is the essence of flakeids.

…

On 14 July 2021 12:20:17 pm AEST, scottmmorris ***@***.***> wrote: The vault id of a cloned vault is currently not preserved between nodes, if it was then this might cause conflicts. For example if a node clones the same vault twice, they would end up with an error as they own two vaults with the same vault id. As in my comment above wouldn't it be better to maintain a copy of the original vault id in the vault map which can be referred to when cloning? -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #208 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[scottmmorris]

Ensuring that all vault IDs are different across keynodes means you have to do bookkeeping on which vault id you are tracking. But is this necessary?

We can make vaultids never conflict by making use of the NodeId. That is the essence of flakeids.

It's necessary in the case above? If a keynode wants to have two separate vaults that are cloned from the same source. I can imagine this might be useful in some cases as we don't provide the option of creating new branches from a single vault yet.

What do you mean by that last part? I'm talking about the case where a keynode has the same vault id stored locally so they would both come under the same node id right?

[CMCDragonkai]

I would keep it simple for now. The vault ids should stay the same, and that should make it easier to pull and get the right remotes too. When pulling from a different remote, we would look for a vault with the same vault id. That would allow us to look up remotes that have the same vault history (or at least expect it to have).

…

On 7/14/21 1:14 PM, scottmmorris wrote: Ensuring that all vault IDs are different across keynodes means you have to do bookkeeping on which vault id you are tracking. But is this necessary? We can make vaultids never conflict by making use of the NodeId. That is the essence of flakeids. It's necessary in the case above? If a keynode wants to have two separate vaults that are cloned from the same source. I can imagine this might be useful in some cases as we don't provide the option of creating new branches from a single vault yet. What do you mean by that last part? I'm talking about the case where a keynode has the same vault id stored locally so they would both come under the same node id right? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#208 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE4OHOM6KDK7QRYLZQ5323TXT6JTANCNFSM5ACHGBNQ>.

[CMCDragonkai]

While we are refactoring the vaults domain and encryptedfs domain:

• https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/205 - vaults review/refactoring
• https://gitlab.com/MatrixAI/Engineering/Polykey/js-encryptedfs/-/merge_requests/46 - encryptedfs review/refactoring

This problem can still be specced out from the UI/UX point of view of the CLI and GUI.

But earnest effort on all of these vault issues will probably be dealt with in a bulk refactoring and layering on new features.

[CMCDragonkai]

• Git merging has lots of edgecases to deal with
• To keep it simple, we can maintain the same vault ID across all the keynodes that are sharing a vault
• Then we should be remembering what is the last node id it was pulled/cloned from, that will automate pulling without needing to specify the node id each time (later this can be generalized to the gestalt)
• When pulling a vault, we need to ensure that the merge is a "fast forward merge", if it is not, the merge is dropped or not completed, this is fine as we want to preserve linear history and simplify the sharing semantics https://www.atlassian.com/git/tutorials/using-branches/git-merge

[CMCDragonkai]

Done by the merging of vaultsrefactoring https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/205

[CMCDragonkai]

Reference docs to be updated here: https://github.com/MatrixAI/js-polykey/issues/258