Merge branch 'master' into rename-config-interface

README.md

@@ -2,10 +2,15 @@
 
 [![CircleCI](https://dl.circleci.com/status-badge/img/gh/entropyxyz/entropy-core/tree/master.svg?style=svg&circle-token=bff4726b78a5f7c7771cb9ee8453cde0b8132d6f)](https://dl.circleci.com/status-badge/redirect/gh/entropyxyz/entropy-core/tree/master)
 
-This repo contains the Entropy blockchain node, the validator server (evaluates programs, stores threshold keyshares, and coordinates threshold-signing), and misc. testing utilities for the network.
+This repo contains the Entropy blockchain node, the [Threshold Signature Server](https://docs-api-entropy-core.vercel.app/entropy_tss) (evaluates programs, stores threshold keyshares, and coordinates threshold-signing), and misc. testing utilities for the network.
 
 Our blockchain node is written with [Substrate](https://substrate.io/) using [Substrate's node template](https://github.com/substrate-developer-hub/substrate-node-template).
 
+## Documentation
+
+- [API documentation](https://docs-api-entropy-core.vercel.app/entropy)
+- [High level documentation for Entropy](https://entropy-docs.vercel.app)
+
 ## Getting Started
 
 You can begin using this repository in a few different ways. This section describes a few of them.
@@ -96,9 +101,15 @@ cargo test -p entropy-tss --features unsafe -- test_sign_tx_no_chain --nocapture
 Once the node template is running locally, you can connect it with **Polkadot-JS Apps** front-end
 to interact with your chain. [Click here](https://polkadot.js.org/apps/#/explorer?rpc=ws://localhost:9944) connecting the Apps to your local node template.
 
+### Command line interface
+
+A [simple command line interface client](https://github.com/entropyxyz/entropy-core/tree/master/crates/test-cli) is included in this repository for test purposes. This can be used with both the local docker-compose network and network deployments.
+
+It is however only intended for use with test networks and has no secure private key storage. For a fully featured command line client see [entropyxyz/cli](https://github.com/entropyxyz/cli).
+
 ## Threshold Keys
 
-- Keys for internal testnet use only, not secure, here for convenience do not use them for anything real
+- Keys for internal devnet use only (used in tests and for the local network built with docker-compose). These are not secure, they are here for convenience, do not use them for anything real.
 
 #### Alice
 

crates/README.md

@@ -1,61 +1,11 @@
-# Crypto
+# Crates
 
-This document serves as a partial spec of this repo.
+This directory contains the [Threshold Signature Server](https://github.com/entropyxyz/entropy-core/tree/master/crates/threshold-signature-server) and some utility crates related to it:
 
-The major actors in this repo are:
-
-- `server`:
-  - `signing-client` - HTTP endpoints run by all nodes for signing protocols
-  - `partition` - Entropy nodes are partitioned into one of $N$ Partitions. Nodes from the same Partition have equivalent secret keyshare information. The user contacts each node from each partition directly with a message containing the share for that partition, by calling `new_user`.
-
-The utility crates in this repo are:
-
-- [`centralized-keygen`](https://github.com/entropyxyz/entropy-core/tree/master/crates/centralized-keygen) - A convenience wrapper around methods from the threshold signing protocol
+- [`shared`](https://github.com/entropyxyz/entropy-core/tree/master/crates/shared) - Common types shared by both the chain node and TSS server
 - [`kvdb`](https://github.com/entropyxyz/entropy-core/tree/master/crates/kvdb) - An encrypted key-value datastore
-- [`entropy-shared`](https://github.com/entropyxyz/entropy-core/tree/master/crates/shared) - Common no-std types, which Substrate requires
+- [`protocol`](https://github.com/entropyxyz/entropy-core/tree/master/crates/protocol) - Transport logic for running the Entropy protocols
 - [`testing-utils`](https://github.com/entropyxyz/entropy-core/tree/master/crates/testing-utils) - Testing utility methods shared across the workspace
+- [`test-cli`](https://github.com/entropyxyz/entropy-core/tree/master/crates/test-cli) - A simple command line client for test purposes
 
 Programs have now moved to [entropyxyz/programs](https://github.com/entropyxyz/programs).
-
-## Documentation of major APIs
-
-## `new_user` - create a new user (HTTP endpoint POST `/user/new`)
-
-<!-- deprecated 2022-08-26: -->
-<!--  -->
-<!-- 1. deprecate: Each Partition Leader is informed of a new user's secret keyshare by the User. User calls `partition_leader::new_user` on each CL. -->
-<!-- 2. Partition Leaders validate that each other CL received a valid keyshare. -->
-<!-- 3. Each CL broadcasts the user's secret keyshare (by calling `new_user`) to each node in their Partition. -->
-<!--  -->
-<!-- Instead: -->
-
-1. The users creates a set of shares, one for each partition, and sends each node in each partition a copy of their partition's share in an encrypted message.
-2. One member of each partition sends a message to the chain to confirm.
-
-<!-- 2. Nodes validate that each other node in their Partition received an identical keyshare. -->
-<!-- 3. Test the share validity: one node from each partition is selected to construct a signature. -->
-<!--  -->
-<!-- - If the signature is valid, end, post (todo: what data) new-user data on chain -->
-<!-- - If the signature is invalid, and no node faulted, user is at fault, fail -->
-<!-- - If the signature is invalid, and a node faulted, slash node, retry with new node from that partition -->
-
-## `sign` - construct a signature to return to the user
-
-1. User submits a transaction (`pallets::relayer::prep_transaction`) to the chain, containing a message including their substrate address and their (hashed) message.
-2. A set of transactions is picked up by the next block proposer (substrate: TODO). The proposed block contains the proposed signing party information (`server/sign_init`).
-3. Upon block creation, signers read the block (`pallets::propagation::post`), containing the IP addresses of nodes who must now execute signing protocols. If a node is in a signing party, it advances to the next step.
-4. The user submits the full message to be signed to nodes in the signing party by making a POST request to `/user/tx`, so that they can check if it meets the configured programs.
-<!-- 2. In the next block, an offchain worker is created (`pallet::propagation::offchain_worker`) -->
-<!--   - currently: by each node. The communication manager waits for calls from each other node about node party information. -->
-<!--   - eventually: after implementing Partitions, the communication manager will already have this information from on-chain. -->
-<!-- 3. CM chooses a signing party (`communication_manager::handle_signing`) -->
-<!-- 4. CM broadcasts the party information, calling `new_party` on each selected signer -->
-5. Each signer calls `subscribe_to_me` on each other signer, subscribing to all party-related messages, creating a an axum SSE of signing-related messages
-6. After each signer has received subscription from each other signer, the nodes proceed to pass signing-protocol related messages until the protocol completes.
-7. If the signing protocol fails, the nodes broadcast information about the faulty signer, to be included in the next block. A subsequent block will designate a replacement signer (TODO: substrate).
-
-## Not yet implemented
-
-Eventually these will also be implemented:
-- `update_node_set` - to be called when the active node-set changes. Updates require resharing stored keyshares to prevent attacks.
-- `delete_user` - remove a user's information from all nodes

crates/kvdb/src/lib.rs

@@ -13,6 +13,7 @@
 // You should have received a copy of the GNU Affero General Public License
 // along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
+//! An encrypted key-value store used for storing keyshares and other private data
 pub mod encrypted_sled;
 pub mod kv_manager;
 use std::{fs, path::PathBuf};

crates/protocol/src/execute_protocol.rs

@@ -80,7 +80,7 @@ async fn execute_protocol_generic<Res: ProtocolResult>(
         // Send out broadcasts
         let destinations = session.broadcast_destinations();
         if let Some(destinations) = destinations {
-            // TODO: this can happen in a spawned task
+            // TODO (#641): this can happen in a spawned task
             let message = session.make_broadcast(&mut OsRng)?;
             for destination in destinations.iter() {
                 tx.send(ProtocolMessage::new(&my_id, destination, message.clone()))?;
@@ -91,7 +91,7 @@ async fn execute_protocol_generic<Res: ProtocolResult>(
         let destinations = session.direct_message_destinations();
         if let Some(destinations) = destinations {
             for destination in destinations.iter() {
-                // TODO: this can happen in a spawned task.
+                // TODO (#641): this can happen in a spawned task.
                 // The artefact will be sent back to the host task
                 // to be added to the accumulator.
                 let (message, artifact) = session.make_direct_message(&mut OsRng, destination)?;
@@ -103,7 +103,7 @@ async fn execute_protocol_generic<Res: ProtocolResult>(
         }
 
         for preprocessed in cached_messages {
-            // TODO: this may happen in a spawned task.
+            // TODO (#641): this may happen in a spawned task.
             let processed = session.process_message(preprocessed)?;
 
             // This will happen in a host task.
@@ -120,7 +120,7 @@ async fn execute_protocol_generic<Res: ProtocolResult>(
                 session.preprocess_message(&mut accum, &message.from, message.payload)?;
 
             if let Some(preprocessed) = preprocessed {
-                // TODO: this may happen in a spawned task.
+                // TODO (#641): this may happen in a spawned task.
                 let result = session.process_message(preprocessed)?;
 
                 // This will happen in a host task.

crates/protocol/src/user/mod.rs

@@ -13,6 +13,8 @@
 // You should have received a copy of the GNU Affero General Public License
 // along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
+//! For private access mode - run the protocol on the client side (the user's device)
+
 #[cfg(feature = "wasm")]
 pub mod wasm;
 use futures::{future, Future};

crates/protocol/src/user/wasm.rs

@@ -13,7 +13,7 @@
 // You should have received a copy of the GNU Affero General Public License
 // along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
-//! Wrappers around functions to run dkg and signing protocols for JS
+//! Wrappers around functions to run DKG and signing protocols for JS
 use js_sys::Error;
 use sp_core::sr25519;
 use subxt::utils::AccountId32;

crates/shared/src/lib.rs

@@ -39,3 +39,6 @@ pub const REFRESHES_PER_SESSION: u32 = 10;
 
 /// Max instructions per wasm program
 pub const MAX_INSTRUCTIONS_PER_PROGRAM: u64 = 100_000;
+
+/// Blocks a transaction is valid for
+pub const MORTALITY_BLOCKS: u64 = 32;

crates/test-cli/README.md

@@ -9,7 +9,7 @@ nodes and two TSS servers.
 
 This could be either:
 
-- A [devnet deployment](https://github.com/entropyxyz/meta/wiki/New-Entropy-network-deployment-runbook), in
+- A [network deployment](https://github.com/entropyxyz/meta/wiki/New-Entropy-network-deployment-runbook), in
 which case you need to specify the endpoint URI. This can be done either by setting the
 `ENTROPY_DEVNET` environment variable or using the `--chain-endpoint` or `-c` command line argument.
 - A [local deployment with docker compose](https://github.com/entropyxyz/meta/wiki/Local-devnet).

crates/test-cli/src/main.rs

@@ -41,6 +41,7 @@ use entropy_testing_utils::{
 use sp_core::{crypto::AccountId32, sr25519, Hasher, Pair};
 use sp_runtime::traits::BlakeTwo256;
 use subxt::{
+    backend::legacy::LegacyRpcMethods,
     utils::{AccountId32 as SubxtAccountId32, H256},
     OnlineClient,
 };
@@ -228,8 +229,9 @@ async fn run_command() -> anyhow::Result<String> {
             let mut programs_info = vec![];
 
             for program in programs {
-                programs_info
-                    .push(Program::from_hash_or_filename(&api, &program_keypair, program).await?.0);
+                programs_info.push(
+                    Program::from_hash_or_filename(&api, &rpc, &program_keypair, program).await?.0,
+                );
             }
 
             let (registered_info, keyshare_option) = register(
@@ -299,7 +301,7 @@ async fn run_command() -> anyhow::Result<String> {
                 None => vec![],
             };
 
-            let hash = store_program(&api, &keypair, program, program_interface).await?;
+            let hash = store_program(&api, &rpc, &keypair, program, program_interface).await?;
             Ok(format!("Program stored {hash}"))
         },
         CliCommand::UpdatePrograms {
@@ -317,8 +319,9 @@ async fn run_command() -> anyhow::Result<String> {
 
             let mut programs_info = Vec::new();
             for program in programs {
-                programs_info
-                    .push(Program::from_hash_or_filename(&api, &program_keypair, program).await?.0);
+                programs_info.push(
+                    Program::from_hash_or_filename(&api, &rpc, &program_keypair, program).await?.0,
+                );
             }
 
             update_programs(
@@ -447,6 +450,7 @@ impl Program {
 
     async fn from_hash_or_filename(
         api: &OnlineClient<EntropyConfig>,
+        rpc: &LegacyRpcMethods<EntropyConfig>,
         keypair: &sr25519::Pair,
         hash_or_filename: String,
     ) -> anyhow::Result<Self> {
@@ -464,17 +468,18 @@ impl Program {
                         };
                         Ok(Self::new(H256(hash_32), configuration))
                     },
-                    Err(_) => Self::from_file(api, keypair, hash_or_filename).await,
+                    Err(_) => Self::from_file(api, rpc, keypair, hash_or_filename).await,
                 }
             },
-            Err(_) => Self::from_file(api, keypair, hash_or_filename).await,
+            Err(_) => Self::from_file(api, rpc, keypair, hash_or_filename).await,
         }
     }
 
     /// Given a path to a .wasm file, read it, store the program if it doesn't already exist, and
     /// return the hash.
     async fn from_file(
         api: &OnlineClient<EntropyConfig>,
+        rpc: &LegacyRpcMethods<EntropyConfig>,
         keypair: &sr25519::Pair,
         filename: String,
     ) -> anyhow::Result<Self> {
@@ -500,7 +505,9 @@ impl Program {
             "If giving an interface description you must also give a configuration"
         );
 
-        match store_program(api, keypair, program_bytecode.clone(), interface_description).await {
+        match store_program(api, rpc, keypair, program_bytecode.clone(), interface_description)
+            .await
+        {
             Ok(hash) => Ok(Self::new(hash, configuration)),
             Err(error) => {
                 if error.to_string().ends_with("ProgramAlreadySet") {