Upgrade Substrate to follow Polkadot releases (#207)

* PR #201 key association logic; to be reverted * Revert "PR #201 key association logic; to be reverted" This reverts commit 44a3ed2. * added types and comments for readability and clarity in staking-extension * subxt points to temp branch with updated deps * refactored server sp_core stuff to use subxt::ext::sp_core * updated server to use subxt 0.24.0 (from 0.20.0) * BROKEN: updated substrate to polkadot-v0.9.30 and jakehemmerle subxt * update Event and Call to RuntimeEvent and RuntimeCall * fix pallet-free-tx weights * fix pallet-transaction-pause weights * fix pallet-staking-extension weights * updated mock runtimes and tests still needs updated free-tx test since FRAME tx are transactional by default * upgraded pallet-free-tx to substrate-polkadot 0.9.30 * pallets are all updated * entropy-runtime tests pass with runtime-benchmarks * handled client rpc and service...? * entropy compiles and tests * additional refactoring * added entropy-executor * entropy tests pass with runtime-benchmarks * fixed testing-utils * thanks jesse, fixed server, everything works * clippy pt 1 * clippy pt 2 * fixing tests * reenable extensions * fixed a test * updated runtime metadata * added babe and grandpa to rpc * jesse rocks; fixed subxt Config * fmt * clippy * taplo * fmt and clippy * clippy * updated deps * updates
entropyxyz · Nov 2, 2022 · 0e31aa7 · 0e31aa7
1 parent 6aca4a2
commit 0e31aa7
Show file tree

Hide file tree

Showing 83 changed files with 5,027 additions and 4,714 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/README.md b/README.md
@@ -204,18 +204,20 @@ by appending your own. A few useful ones are as follow.
 ```
 
 ## testnet
- * Currently our network requires 2 binaries
- * ``` cargo build --release ``` will build both
- * to run both you can reference /scrips.alice.sh for the chain and /scripts/sig_client.sh for the threshold client
- * the sig client requires a mneumonic as an env and the one in the /scripts/sig_client.sh file is already hardcoded into the chain config (best to use that one)
+
+- Currently our network requires 2 binaries
+- ``` cargo build --release ``` will build both
+- to run both you can reference /scrips.alice.sh for the chain and /scripts/sig_client.sh for the threshold client
+- the sig client requires a mneumonic as an env and the one in the /scripts/sig_client.sh file is already hardcoded into the chain config (best to use that one)
 
 ### changing defaults
-* all defaults are ready to go out the box but can be changed if needed with varying degrees of difficult
-* to change chain address away from default ws://127.0.0.1:9944 you need to inform the sig client which can be done with the env variable ```export ENDPOINT=```
-* To change the default of the sig client from ```http://127.0.0.1:3001/sign``` you need to tell the chain after it is running by making an rpc call. Example code can be found here ```https://github.com/Entropyxyz/util-scripts/blob/master/setEndpoint.ts```. You also need to maintain the route as /sign
 
+* all defaults are ready to go out the box but can be changed if needed with varying degrees of difficult
+- to change chain address away from default ws://127.0.0.1:9944 you need to inform the sig client which can be done with the env variable ```export ENDPOINT=```
+- To change the default of the sig client from ```http://127.0.0.1:3001/sign``` you need to tell the chain after it is running by making an rpc call. Example code can be found here ```https://github.com/entropyxyz/util-scripts/blob/master/setEndpoint.ts```. You also need to maintain the route as /sign
 
 ## Threshold keys
+
 * keys for internal testnet use only, not secure, here for convience do not use them for anything real
 
 Alice
@@ -234,43 +236,42 @@ Secret phrase `where sight patient orphan general short empower hope party hurt
   Account ID:         0x2a8200850770290c7ea3b50a8ff64c6761c882ff8393dc95fccb5d1475eff17f
   SS58 Address:       5D2SVCUkK5FgFiBwPTJuTN65J6fACSEoZrL41thZBAycwnQV
 
-
 ## Running Devnet
-* devnet requires 2 validator nodes, 2 threshold clients running on the same machine
 
-* open 5 terminals lol
+* devnet requires 2 validator nodes, 2 threshold clients running on the same machine
 
-* In terminal 1 set up chain 1
-  * ```cargo build --release```
-  * ```./scripts/alice.sh```
+- open 5 terminals lol
 
-* In terminal 2 run alice threshold client
-  * ```cargo build --release --features="alice unsafe"```
-  * ```./scripts/server.sh```
+- In terminal 1 set up chain 1
+  - ```cargo build --release```
+  - ```./scripts/alice.sh```
 
-* In termainl 3 run chain 2
-  * ```./scripts/bob.sh```
+- In terminal 2 run alice threshold client
+  - ```cargo build --release --features="alice unsafe"```
+  - ```./scripts/server.sh```
 
-* In termainl 5run bob threshold client
-  * ```cargo build --release --features="bob unsafe"```
-  * ```./scripts/server_bob.sh```
+- In termainl 3 run chain 2
+  - ```./scripts/bob.sh```
 
+- In termainl 5run bob threshold client
+  - ```cargo build --release --features="bob unsafe"```
+  - ```./scripts/server_bob.sh```
 
 With all 4 nodes running the chain is now working, next we now have a clash where both chains by default send their OCW messages to port 3001, you need to change one of those
 
-* from this repo https://github.com/entropyxyz/util-scripts
-  * need to setup the repo and link the wasm first
-  * ```cd pkg```
-  * ```npm link```
-  * ```cd ..```
-  * ```npm link x25519-chacha20poly1305-wasm```
-* run setEndpoint.ts
-  * ```ts-node setEndpoint.ts```
-
+- from this repo <https://github.com/entropyxyz/util-scripts>
+  - need to setup the repo and link the wasm first
+  - ```cd pkg```
+  - ```npm link```
+  - ```cd ..```
+  - ```npm link x25519-chacha20poly1305-wasm```
+- run setEndpoint.ts
+  - ```ts-node setEndpoint.ts```
 
 next register
-  * ```ts-node register.ts```
+
+- ```ts-node register.ts```
 
 now you can sign
-  * ```ts-node sign.ts```
 
+- ```ts-node sign.ts```
diff --git a/crypto/centralized-keygen/README.md b/crypto/centralized-keygen/README.md
@@ -1,2 +1,2 @@
 # Protocol Implementation
-A convenience wrapper around methods from our [tofn](https://github.com/Entropyxyz/tofn) fork.
+A convenience wrapper around methods from our [tofn](https://github.com/entropyxyz/tofn) fork.
diff --git a/crypto/centralized-keygen/src/lib.rs b/crypto/centralized-keygen/src/lib.rs
@@ -5,7 +5,7 @@ use tofn::gg20::ceygen;
 
 /// Split a `secret_key` into `threshold`-of-`parties` shards, write to directory `path`.
 /// A wrapper around entropyxyz/tofn.
-/// See https://github.com/Entropyxyz/tofn for details.
+/// See https://github.com/entropyxyz/tofn for details.
 pub fn ceygen(
     path: PathBuf,
     parties: usize,

diff --git a/crypto/kvdb/Cargo.toml b/crypto/kvdb/Cargo.toml
@@ -13,7 +13,7 @@ thiserror   ="1.0.31"
 project-root="0.2.2"
 
 # Crypto
-tofn            ={ git="https://github.com/Entropyxyz/tofn", branch="main" }
+tofn            ={ git="https://github.com/entropyxyz/tofn", branch="main" }
 zeroize         ={ version="1.4", features=["zeroize_derive"], default-features=false }
 rpassword       ={ version="5.0", default-features=false }
 scrypt          ={ version="0.8", default-features=false, features=["std"] }

diff --git a/crypto/server/Cargo.toml b/crypto/server/Cargo.toml
@@ -24,15 +24,15 @@ futures="0.3"
 tokio  ={ version="1.16", features=["macros", "fs", "rt-multi-thread", "io-util"] }
 
 # HTTP
-rocket ={ version="0.5.0-rc.1", default-features=false, features=["json"] }
+rocket ={ version="0.5.0-rc.2", default-features=false, features=["json"] }
 reqwest={ version="0.11", features=["json", "stream"] }
 
 # Substrate
+subxt             ={ package="subxt", git="https://github.com/entropyxyz/subxt.git", branch="substrate-master" }
 parity-scale-codec="3.0.0"
-subxt             ={ git="https://github.com/paritytech/subxt.git" }
-sp-keyring        ="6.0.0"
-sp-core           ="6.0.0"
-tofn              ={ git="https://github.com/Entropyxyz/tofn", branch="main" }
+sp-keyring        ={ package="sp-keyring", git="https://github.com/paritytech/substrate", branch="polkadot-v0.9.30" }
+sp-core           ={ package="sp-core", git="https://github.com/paritytech/substrate", branch="polkadot-v0.9.30" }
+tofn              ={ git="https://github.com/entropyxyz/tofn", branch="main" }
 
 # Entropy
 substrate-common={ path="../substrate-common", default-features=false }

diff --git a/crypto/server/entropy_metadata.scale b/crypto/server/entropy_metadata.scale
diff --git a/crypto/server/src/chain_api.rs b/crypto/server/src/chain_api.rs
@@ -1,14 +1,16 @@
 #![allow(clippy::all)]
-use subxt::{ClientBuilder, DefaultConfig, PairSigner, PolkadotExtrinsicParams};
+pub use subxt::config::PolkadotConfig as EntropyConfig;
+use subxt::{
+    config::Config,
+    tx::{PairSigner, SubstrateExtrinsicParams},
+    OnlineClient,
+};
 #[subxt::subxt(runtime_metadata_path = "entropy_metadata.scale")]
 pub mod entropy {}
 
-pub type EntropyRuntime =
-    entropy::RuntimeApi<DefaultConfig, PolkadotExtrinsicParams<DefaultConfig>>;
-
 /// Creates an api instance to talk to chain
 /// Chain endpoint set on launch
-pub async fn get_api(url: &str) -> Result<EntropyRuntime, subxt::Error<entropy::DispatchError>> {
-    let api = ClientBuilder::new().set_url(url).build().await?.to_runtime_api::<EntropyRuntime>();
+pub async fn get_api(url: &str) -> Result<OnlineClient<EntropyConfig>, subxt::Error> {
+    let api = OnlineClient::<EntropyConfig>::from_url(url).await?;
     Ok(api)
 }
diff --git a/crypto/server/src/main.rs b/crypto/server/src/main.rs
@@ -29,9 +29,9 @@ use bip39::{Language, Mnemonic, MnemonicType};
 extern crate rocket;
 use kvdb::kv_manager::{error::KvError, KeyReservation, KvManager};
 use rocket::routes;
-use sp_core::{crypto::AccountId32, sr25519, Pair};
 use sp_keyring::AccountKeyring;
 use substrate_common::SIGNING_PARTY_SIZE;
+use subxt::ext::sp_core::{crypto::AccountId32, sr25519, Pair};
 
 use self::{
     signing_client::{api::*, SignerState},

diff --git a/crypto/server/src/message.rs b/crypto/server/src/message.rs
@@ -7,8 +7,8 @@ use chacha20poly1305::{
 use rand_core::OsRng;
 use rocket::serde::json::to_string;
 use serde::{Deserialize, Serialize};
-use sp_core::{crypto::AccountId32, sr25519, sr25519::Signature, Bytes, Pair};
 use sp_keyring::AccountKeyring;
+use subxt::ext::sp_core::{crypto::AccountId32, sr25519, sr25519::Signature, Bytes, Pair};
 use x25519_dalek::{PublicKey, StaticSecret};
 use zeroize::Zeroize;
 
@@ -60,7 +60,7 @@ impl SignedMessage {
         Ok(SignedMessage {
             pk: sk.public().0,
             a: *a.as_bytes(),
-            msg: sp_core::Bytes(ciphertext),
+            msg: subxt::ext::sp_core::Bytes(ciphertext),
             nonce: static_nonce,
             sig: sk.sign(&hash),
             recip: recip.to_bytes(),
@@ -179,7 +179,7 @@ mod tests {
         // Test encryption & signing.
         let encrypt_result = SignedMessage::new(&alice, &plaintext, &bob_public_key);
         // Assert no error received in encryption.
-        assert!(!encrypt_result.is_err());
+        assert!(encrypt_result.is_ok());
         let encrypted_message = encrypt_result.unwrap();
 
         // Test signature validity
@@ -188,7 +188,7 @@ mod tests {
         // Test decryption
         let decrypt_result = encrypted_message.decrypt(&bob);
         // Assert no error received in decryption.
-        assert!(!decrypt_result.is_err());
+        assert!(decrypt_result.is_ok());
         let decrypted_result = decrypt_result.unwrap();
 
         // Check the decrypted message equals the plaintext.

diff --git a/crypto/server/src/signing_client/api.rs b/crypto/server/src/signing_client/api.rs
@@ -1,11 +1,11 @@
-use std::str;
+use std::{io, str};
 
 use k256::ecdsa::recoverable;
 use kvdb::kv_manager::KvManager;
 use parity_scale_codec::Decode;
 use rocket::{http::Status, response::stream::EventStream, serde::json::Json, Shutdown, State};
 use substrate_common::OCWMessage;
-use subxt::sp_runtime::AccountId32;
+use subxt::ext::sp_runtime::AccountId32;
 use tofn::sdk::api::Signature;
 use tracing::instrument;
 
@@ -81,7 +81,6 @@ pub async fn new_party(
 
 /// Other nodes in the party call this method to subscribe to this node's broadcasts.
 /// The SigningProtocol begins when all nodes in the party have called this method on this node.
-#[instrument]
 #[post("/subscribe_to_me", data = "<msg>")]
 pub async fn subscribe_to_me(
     msg: Json<SubscribeMessage>,

diff --git a/crypto/server/src/signing_client/tests.rs b/crypto/server/src/signing_client/tests.rs
@@ -21,7 +21,7 @@ use serial_test::serial;
 use sp_core::{crypto::AccountId32, sr25519::Pair as Sr25519Pair, Pair as Pair2};
 use sp_keyring::AccountKeyring;
 use substrate_common::{Message, SigRequest};
-use subxt::{sp_core::sr25519, PairSigner};
+use subxt::{ext::sp_core::sr25519, tx::PairSigner};
 use testing_utils::context::{test_context, test_context_stationary};
 use tofn::{
     gg20::keygen::{KeygenPartyId, SecretKeyShare},
@@ -152,7 +152,7 @@ async fn create_clients(port: i64, key_number: String) -> Rocket<Ignite> {
 
     // Shortcut: store the shares manually
     let root = project_root::get_project_root().unwrap();
-    let share_id = if port == 3001 { 0 } else { 1 };
+    let share_id = i32::from(port != 3001);
     let path: PathBuf =
         [root, "test_data".into(), "key_shares".into(), share_id.to_string().into()]
             .into_iter()

diff --git a/crypto/server/src/user/api.rs b/crypto/server/src/user/api.rs
@@ -6,15 +6,21 @@ use kvdb::kv_manager::{
 };
 use log::info;
 use rocket::{http::Status, response::stream::EventStream, serde::json::Json, Shutdown, State};
-use sp_core::{sr25519, Pair};
 use substrate_common::SIGNING_PARTY_SIZE;
-use subxt::{sp_runtime::AccountId32, DefaultConfig, PairSigner};
+use subxt::{
+    ext::{
+        sp_core::{sr25519, Pair},
+        sp_runtime::AccountId32,
+    },
+    tx::PairSigner,
+    OnlineClient,
+};
 use tracing::instrument;
 use zeroize::Zeroize;
 
 use super::{ParsedUserInputPartyInfo, UserErr, UserInputPartyInfo};
 use crate::{
-    chain_api::{entropy, get_api, EntropyRuntime},
+    chain_api::{entropy, get_api, EntropyConfig},
     message::SignedMessage,
     signing_client::SignerState,
     Configuration,
@@ -49,7 +55,7 @@ pub async fn new_user(
             // store new user data in kvdb
             let subgroup = get_subgroup(&api, &signer)
                 .await?
-                .ok_or(UserErr::SubgroupError("Subgroup Error"))?;
+                .ok_or_else(|| UserErr::SubgroupError("Subgroup Error"))?;
             let reservation = state.kv().reserve_key(key.to_string()).await?;
             state.kv().put(reservation, v).await?;
             // TODO: Error handling really complex needs to be thought about.
@@ -62,28 +68,29 @@ pub async fn new_user(
     Ok(Status::Ok)
 }
 
-pub async fn is_registering(api: &EntropyRuntime, who: &AccountId32) -> Result<bool, UserErr> {
-    let is_registering = api
-        .storage()
-        .relayer()
-        .registering(who, None)
-        .await?
-        .ok_or(UserErr::NotRegistering("Register Onchain first"))?;
-    Ok(is_registering.is_registering)
+pub async fn is_registering(
+    api: &OnlineClient<EntropyConfig>,
+    who: &AccountId32,
+) -> Result<bool, UserErr> {
+    let is_registering_query = entropy::storage().relayer().registering(who);
+    let is_registering = api.storage().fetch(&is_registering_query, None).await.unwrap();
+    Ok(is_registering
+        .ok_or_else(|| UserErr::NotRegistering("Register Onchain first"))?
+        .is_registering)
 }
 
 // Returns PairSigner for this nodes threshold server.
 // The PairSigner is stored as an encrypted mnemonic in the kvdb and
 // is used for PKE and to submit extrensics on chain.
 pub async fn get_signer(
     kv: &KvManager,
-) -> Result<subxt::PairSigner<DefaultConfig, sr25519::Pair>, KvError> {
+) -> Result<PairSigner<EntropyConfig, sr25519::Pair>, KvError> {
     let exists = kv.kv().exists("MNEMONIC").await?;
     let raw_m = kv.kv().get("MNEMONIC").await?;
     match core::str::from_utf8(&raw_m) {
         Ok(s) => match Mnemonic::from_phrase(s, Language::English) {
             Ok(m) => match <sr25519::Pair as Pair>::from_phrase(m.phrase(), None) {
-                Ok(p) => Ok(PairSigner::<DefaultConfig, sr25519::Pair>::new(p.0)),
+                Ok(p) => Ok(PairSigner::<EntropyConfig, sr25519::Pair>::new(p.0)),
                 Err(e) => Err(KvError::GetErr(InnerKvError::LogicalErr("SENSITIVE".to_owned()))),
             },
             Err(e) => Err(KvError::GetErr(InnerKvError::LogicalErr(e.to_string()))),
@@ -93,19 +100,20 @@ pub async fn get_signer(
 }
 
 pub async fn get_subgroup(
-    api: &EntropyRuntime,
-    signer: &subxt::PairSigner<DefaultConfig, sr25519::Pair>,
+    api: &OnlineClient<EntropyConfig>,
+    signer: &PairSigner<EntropyConfig, sr25519::Pair>,
 ) -> Result<Option<u8>, UserErr> {
     let mut subgroup: Option<u8> = None;
     let address = signer.account_id();
     // TODO: stash keys are broken up into subgroups....need to get stash key here from threshold
     for i in 0..SIGNING_PARTY_SIZE {
+        let signing_group_addresses_query =
+            entropy::storage().staking_extension().signing_groups(i as u8);
         let signing_group_addresses = api
             .storage()
-            .staking_extension()
-            .signing_groups(&(i as u8), None)
+            .fetch(&signing_group_addresses_query, None)
             .await?
-            .ok_or(UserErr::SubgroupError("Subgroup Error"))?;
+            .ok_or_else(|| UserErr::SubgroupError("Subgroup Error"))?;
         if signing_group_addresses.contains(address) {
             subgroup = Some(i as u8);
             break;
@@ -115,19 +123,23 @@ pub async fn get_subgroup(
 }
 
 pub async fn confirm_registered(
-    api: &EntropyRuntime,
+    api: &OnlineClient<EntropyConfig>,
     who: AccountId32,
     subgroup: u8,
-    signer: &subxt::PairSigner<DefaultConfig, sr25519::Pair>,
-) -> Result<(), subxt::Error<entropy::DispatchError>> {
+    signer: &PairSigner<EntropyConfig, sr25519::Pair>,
+) -> Result<(), subxt::error::Error> {
     // TODO error handling + return error
     // TODO fire and forget, or wait for in block maybe Ddos error
-    let _ = api.tx().relayer()
-        .confirm_register(who, subgroup)
-        // TODO: Understand this better, potentially use sign_and_submit_default
-        // or other method under sign_and_*
-        .sign_and_submit_then_watch_default(signer).await?
-        .wait_for_in_block().await?
-        .wait_for_success().await?;
+    // TODO: Understand this better, potentially use sign_and_submit_default
+    // or other method under sign_and_*
+    let registration_tx = entropy::tx().relayer().confirm_register(who, subgroup);
+    let _ = api
+        .tx()
+        .sign_and_submit_then_watch_default(&registration_tx, signer)
+        .await?
+        .wait_for_in_block()
+        .await?
+        .wait_for_success()
+        .await?;
     Ok(())
 }
diff --git a/crypto/server/src/user/errors.rs b/crypto/server/src/user/errors.rs
@@ -20,9 +20,9 @@ pub enum UserErr {
     #[error("Kv error: {0}")]
     Kv(#[from] kvdb::kv_manager::error::KvError),
     #[error("Substrate error: {0}")]
-    Substrate(#[from] subxt::Error<entropy::DispatchError>),
+    Substrate(#[from] subxt::error::DispatchError),
     #[error("Generic Substrate error: {0}")]
-    GenericSubstrate(#[from] subxt::GenericError<std::convert::Infallible>),
+    GenericSubstrate(#[from] subxt::error::Error),
     #[error("Not Registering error: {0}")]
     NotRegistering(&'static str),
     #[error("Subgroup error: {0}")]