Utilize Metadata V15 (#1041)

* Update frame-metadata to the latest branch Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Add outer enum types Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Extend the extrinsic with address,call,sign,extra types Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Codegen test Event, Error and Call for outer enums Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Revert "Codegen test Event, Error and Call for outer enums" This reverts commit db542dc. * Update frame-metadata from the latest release Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update scale-info Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen/error: Support v15 message Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Convert v14 to v15 Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata/retain: Adjust to extrinsic type for V15 Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata/validation: Adjust hashing for extrinsic types V15 Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * scripts: Fetch V15 and output codegen for full_client only Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt/blocks: Use extrinsic types directly Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * testing: Fetch V15 for build script Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * artifacts: Generate from latest polkadot version Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen: Fetch legacy with old API for v14 only Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * rpc: Fetch metadata versions Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * client: Fetch latest unstable then V15 then V14 Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * testing: Adjust testing API to latest interface Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Adjust clippy Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Generate the `RuntimeError` type for V14 Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Remove testing files Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * testing/staking: Remove controller account from bond Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata/validation: Use specific variants for hashing RuntimeCall Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * XXX: Custom Substrate binary: must revert with next release Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * XXX: To revert: CI use hardcoded substrate Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen: Use v15 outer enum types Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Retain outer enum types Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * codegen: Use outer enum types instead of generating them Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update artifacts Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Revert "XXX: Custom Substrate binary: must revert with next release" This reverts commit e970529. Revert "XXX: To revert: CI use hardcoded substrate" This reverts commit b18a5a0. * testing: Include env for dummy wat contracts Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Adjsut clippy Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * ci: Use new link for fetching latest substrate binary Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * tests: Include dummy RuntimeEvent into test metadata Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * ci: Bump light-client timeout tests to 25min Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata/validation: Use specific pallets as provided Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * testing: Rename metadata constant Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Use call_ty instead of signature_ty Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Rename retaining variant function Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Use Option<&[&str]> Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * online_client: Fetch V15 metadata explicitely Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata/validation: Include the hash of the outer enum types Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Fix sign typo Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * artifacts: Update the artifacts Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Remove RootError RootEvent and RootExtrinsic traits Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update polkadot.rs Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata/tests: Ensure outer enum variants are retained Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * scripts: Include multiple pallets for our decoding purposes Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Apply clippy Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * artifacts: Update small metadata Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * error: Keep raw bytes for the ModuleError representation Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * error: Modify docs to not include links Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt/tests: Propagate `RuntimeCall` to outer enums Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * subxt: Provide proper byte slice for decoding Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update artifacts Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * cli/tests: Adjust expected pallets message Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Test conversion from v14 to v15 Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata: Fix typo Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Update subxt/src/blocks/extrinsic_types.rs Co-authored-by: James Wilson <james@jsdw.me> * metadata: Simplify type path for RuntimeError Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * metadata/validation: Use visited ids per outer enum Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * error: Remove RawModuleError Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> * Fix new clippy error from updated rust version Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: James Wilson <james@jsdw.me>
paritytech · Jul 17, 2023 · 78a106f · 78a106f
1 parent 4fb051f
commit 78a106f
Show file tree

Hide file tree

Showing 33 changed files with 3,251 additions and 9,024 deletions.
diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml
@@ -9,7 +9,7 @@ on:
 env:
   CARGO_TERM_COLOR: always
   # Use latest substrate for nightly runs:
-  SUBSTRATE_URL: https://releases.parity.io/substrate/x86_64-debian:stretch/latest/substrate/substrate
+  SUBSTRATE_URL: https://releases.parity.io/substrate/x86_64-debian:bullseye/latest/substrate/substrate
 
 jobs:
   tests:

diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
@@ -19,7 +19,7 @@ concurrency:
 env:
     CARGO_TERM_COLOR: always
     # TODO: Currently pointing at latest substrate; is there a suitable binary we can pin to here?
-    SUBSTRATE_URL: https://releases.parity.io/substrate/x86_64-debian:stretch/latest/substrate/substrate
+    SUBSTRATE_URL: https://releases.parity.io/substrate/x86_64-debian:bullseye/latest/substrate/substrate
 
 jobs:
     build:

diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -45,7 +45,7 @@ console_error_panic_hook = "0.1.7"
 darling = "0.20.0"
 derivative = "2.2.0"
 either = "1.8.1"
-frame-metadata = { version = "15.1.0", features = ["v14", "v15-unstable", "std"] }
+frame-metadata = { version = "16.0.0", default-features = false, features = ["current", "std"] }
 futures = { version = "0.3.27", default-features = false, features = ["std"] }
 getrandom = { version = "0.2", default-features = false }
 hex = "0.4.3"
@@ -58,7 +58,7 @@ proc-macro-error = "1.0.4"
 proc-macro2 = "1.0.63"
 quote = "1.0.29"
 regex = "1.9.1"
-scale-info = "2.8.0"
+scale-info = "2.9.0"
 scale-value = "0.10.0"
 scale-bits = "0.3"
 scale-decode = "0.7.0"

diff --git a/artifacts/polkadot_metadata_full.scale b/artifacts/polkadot_metadata_full.scale
diff --git a/artifacts/polkadot_metadata_small.scale b/artifacts/polkadot_metadata_small.scale
diff --git a/artifacts/polkadot_metadata_tiny.scale b/artifacts/polkadot_metadata_tiny.scale
diff --git a/cli/src/commands/explore/mod.rs b/cli/src/commands/explore/mod.rs
@@ -167,7 +167,7 @@ pub mod tests {
     async fn test_commands() {
         // show pallets:
         let output = simulate_run("").await;
-        assert_eq!(output.unwrap(), "Usage:\n    subxt explore <PALLET>\n        explore a specific pallet\n\nAvailable <PALLET> values are:\n    Balances\n    Multisig\n    Staking\n    System\n");
+        assert_eq!(output.unwrap(), "Usage:\n    subxt explore <PALLET>\n        explore a specific pallet\n\nAvailable <PALLET> values are:\n    Balances\n    Multisig\n    ParaInherent\n    Staking\n    System\n    Timestamp\n");
         // if incorrect pallet, error:
         let output = simulate_run("abc123").await;
         assert!(output.is_err());

diff --git a/codegen/src/api/mod.rs b/codegen/src/api/mod.rs
@@ -289,13 +289,12 @@ impl RuntimeGenerator {
     ) -> Result<TokenStream2, CodegenError> {
         let item_mod_attrs = item_mod.attrs.clone();
         let item_mod_ir = ir::ItemMod::try_from(item_mod)?;
-        let default_derives = derives.default_derives();
 
         let type_gen = TypeGenerator::new(
             self.metadata.types(),
             "runtime_types",
             type_substitutes,
-            derives.clone(),
+            derives,
             crate_path.clone(),
             should_gen_docs,
         );
@@ -382,118 +381,6 @@ impl RuntimeGenerator {
             })
             .collect::<Result<Vec<_>, CodegenError>>()?;
 
-        let outer_event_variants = self.metadata.pallets().filter_map(|p| {
-            let variant_name = format_ident!("{}", p.name());
-            let mod_name = format_ident!("{}", p.name().to_string().to_snake_case());
-            let index = proc_macro2::Literal::u8_unsuffixed(p.index());
-
-            p.event_ty_id().map(|_| {
-                quote! {
-                    #[codec(index = #index)]
-                    #variant_name(#mod_name::Event),
-                }
-            })
-        });
-
-        let outer_event = quote! {
-            #default_derives
-            pub enum Event {
-                #( #outer_event_variants )*
-            }
-        };
-
-        let outer_extrinsic_variants = self.metadata.pallets().filter_map(|p| {
-            let variant_name = format_ident!("{}", p.name());
-            let mod_name = format_ident!("{}", p.name().to_string().to_snake_case());
-            let index = proc_macro2::Literal::u8_unsuffixed(p.index());
-
-            p.call_ty_id().map(|_| {
-                quote! {
-                    #[codec(index = #index)]
-                    #variant_name(#mod_name::Call),
-                }
-            })
-        });
-
-        let outer_extrinsic = quote! {
-            #default_derives
-            pub enum Call {
-                #( #outer_extrinsic_variants )*
-            }
-        };
-
-        let root_event_if_arms = self.metadata.pallets().filter_map(|p| {
-            let variant_name_str = &p.name();
-            let variant_name = format_ident!("{}", variant_name_str);
-            let mod_name = format_ident!("{}", variant_name_str.to_string().to_snake_case());
-
-            p.event_ty_id().map(|_| {
-                // An 'if' arm for the RootEvent impl to match this variant name:
-                quote! {
-                    if pallet_name == #variant_name_str {
-                        return Ok(Event::#variant_name(#mod_name::Event::decode_with_metadata(
-                            &mut &*pallet_bytes,
-                            pallet_ty,
-                            metadata
-                        )?));
-                    }
-                }
-            })
-        });
-
-        let root_extrinsic_if_arms = self.metadata.pallets().filter_map(|p| {
-            let variant_name_str = p.name();
-            let variant_name = format_ident!("{}", variant_name_str);
-            let mod_name = format_ident!("{}", variant_name_str.to_string().to_snake_case());
-            p.call_ty_id().map(|_| {
-                // An 'if' arm for the RootExtrinsic impl to match this variant name:
-                quote! {
-                    if pallet_name == #variant_name_str {
-                        return Ok(Call::#variant_name(#mod_name::Call::decode_with_metadata(
-                            &mut &*pallet_bytes,
-                            pallet_ty,
-                            metadata
-                        )?));
-                    }
-                }
-            })
-        });
-
-        let outer_error_variants = self.metadata.pallets().filter_map(|p| {
-            let variant_name = format_ident!("{}", p.name());
-            let mod_name = format_ident!("{}", p.name().to_string().to_snake_case());
-            let index = proc_macro2::Literal::u8_unsuffixed(p.index());
-
-            p.error_ty_id().map(|_| {
-                quote! {
-                    #[codec(index = #index)]
-                    #variant_name(#mod_name::Error),
-                }
-            })
-        });
-
-        let outer_error = quote! {
-            #default_derives
-            pub enum Error {
-                #( #outer_error_variants )*
-            }
-        };
-
-        let root_error_if_arms = self.metadata.pallets().filter_map(|p| {
-            let variant_name_str = &p.name();
-            let variant_name = format_ident!("{}", variant_name_str);
-            let mod_name = format_ident!("{}", variant_name_str.to_string().to_snake_case());
-
-            p.error_ty_id().map(|type_id| {
-                quote! {
-                    if pallet_name == #variant_name_str {
-                        let variant_error = #mod_name::Error::decode_with_metadata(cursor, #type_id, metadata)?;
-                        return Ok(Error::#variant_name(variant_error));
-                    }
-                }
-            })
-        });
-
         let mod_ident = &item_mod_ir.ident;
         let pallets_with_constants: Vec<_> = pallets_with_mod_names
             .iter()
@@ -526,6 +413,12 @@ impl RuntimeGenerator {
             should_gen_docs,
         )?;
 
+        // Fetch the paths of the outer enums.
+        // Substrate exposes those under `kitchensink_runtime`, while Polkadot under `polkadot_runtime`.
+        let call_path = type_gen.resolve_type_path(self.metadata.outer_enums().call_enum_ty());
+        let event_path = type_gen.resolve_type_path(self.metadata.outer_enums().event_enum_ty());
+        let error_path = type_gen.resolve_type_path(self.metadata.outer_enums().error_enum_ty());
+
         Ok(quote! {
             #( #item_mod_attrs )*
             #[allow(dead_code, unused_imports, non_camel_case_types)]
@@ -551,36 +444,14 @@ impl RuntimeGenerator {
                 /// The error type returned when there is a runtime issue.
                 pub type DispatchError = #types_mod_ident::sp_runtime::DispatchError;
 
-                #outer_event
+                /// The outer event enum.
+                pub type Event = #event_path;
 
-                impl #crate_path::events::RootEvent for Event {
-                    fn root_event(pallet_bytes: &[u8], pallet_name: &str, pallet_ty: u32, metadata: &#crate_path::Metadata) -> Result<Self, #crate_path::Error> {
-                        use #crate_path::metadata::DecodeWithMetadata;
-                        #( #root_event_if_arms )*
-                        Err(#crate_path::ext::scale_decode::Error::custom(format!("Pallet name '{}' not found in root Event enum", pallet_name)).into())
-                    }
-                }
-
-                #outer_extrinsic
+                /// The outer extrinsic enum.
+                pub type Call = #call_path;
 
-                impl #crate_path::blocks::RootExtrinsic for Call {
-                    fn root_extrinsic(pallet_bytes: &[u8], pallet_name: &str, pallet_ty: u32, metadata: &#crate_path::Metadata) -> Result<Self, #crate_path::Error> {
-                        use #crate_path::metadata::DecodeWithMetadata;
-                        #( #root_extrinsic_if_arms )*
-                        Err(#crate_path::ext::scale_decode::Error::custom(format!("Pallet name '{}' not found in root Call enum", pallet_name)).into())
-                    }
-                }
-
-                #outer_error
-
-                impl #crate_path::error::RootError for Error {
-                    fn root_error(pallet_bytes: &[u8], pallet_name: &str, metadata: &#crate_path::Metadata) -> Result<Self, #crate_path::Error> {
-                        use #crate_path::metadata::DecodeWithMetadata;
-                        let cursor = &mut &pallet_bytes[..];
-                        #( #root_error_if_arms )*
-                        Err(#crate_path::ext::scale_decode::Error::custom(format!("Pallet name '{}' not found in root Error enum", pallet_name)).into())
-                    }
-                }
+                /// The outer error enum representing the DispatchError's Module variant.
+                pub type Error = #error_path;
 
                 pub fn constants() -> ConstantsApi {
                     ConstantsApi

diff --git a/codegen/src/error.rs b/codegen/src/error.rs
@@ -19,7 +19,7 @@ pub enum CodegenError {
     #[error("Failed IO for {0}, make sure that you are providing the correct file path for metadata: {1}")]
     Io(String, std::io::Error),
     /// Cannot decode the metadata bytes.
-    #[error("Could not decode metadata, only V14 metadata is supported: {0}")]
+    #[error("Could not decode metadata, only V14 and V15 metadata are supported: {0}")]
     Decode(#[from] codec::Error),
     /// Out of line modules are not supported.
     #[error("Out-of-line subxt modules are not supported, make sure you are providing a body to your module: pub mod polkadot {{ ... }}")]

diff --git a/codegen/src/utils/fetch_metadata.rs b/codegen/src/utils/fetch_metadata.rs
@@ -204,12 +204,9 @@ async fn fetch_metadata(
         client: &impl ClientT,
         version: MetadataVersion,
     ) -> Result<Vec<u8>, FetchMetadataError> {
-        if !matches!(
-            version,
-            MetadataVersion::Latest | MetadataVersion::Version(14)
-        ) {
+        if !matches!(version, MetadataVersion::Version(14)) {
             return Err(FetchMetadataError::Other(
-                "The node can only return version 14 metadata but you've asked for something else"
+                "The node can only return version 14 metadata using the legacy API but you've asked for something else"
                     .to_string(),
             ));
         }

diff --git a/metadata/Cargo.toml b/metadata/Cargo.toml
@@ -24,6 +24,7 @@ thiserror = { workspace = true }
 bitvec = { workspace = true, features = ["alloc"] }
 criterion = { workspace = true }
 scale-info = { workspace = true, features = ["bit-vec"] }
+assert_matches = { workspace = true }
 
 [lib]
 # Without this, libtest cli opts interfere with criteron benches: