Make the codegen more accessible for running in WASM (#1154)

* minor api changes * parsing instead of format_ident! * fix self issue * expose types on typegen * web wasm support via feature flag * fmt and clippy * small adjustments * remove exposing of types() in type generator * adjust compile error * remove little any flag * Need access to the type registry from the typegen again * Bump proc-macro2 from 1.0.66 to 1.0.67 (#1164) Bumps [proc-macro2](https://github.com/dtolnay/proc-macro2) from 1.0.66 to 1.0.67. - [Release notes](https://github.com/dtolnay/proc-macro2/releases) - [Commits](dtolnay/proc-macro2@1.0.66...1.0.67) --- updated-dependencies: - dependency-name: proc-macro2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump serde_json from 1.0.106 to 1.0.107 (#1166) Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.106 to 1.0.107. - [Release notes](https://github.com/serde-rs/json/releases) - [Commits](serde-rs/json@v1.0.106...v1.0.107) --- updated-dependencies: - dependency-name: serde_json dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump trybuild from 1.0.84 to 1.0.85 (#1167) Bumps [trybuild](https://github.com/dtolnay/trybuild) from 1.0.84 to 1.0.85. - [Release notes](https://github.com/dtolnay/trybuild/releases) - [Commits](dtolnay/trybuild@1.0.84...1.0.85) --- updated-dependencies: - dependency-name: trybuild dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump Swatinem/rust-cache from 2.6.2 to 2.7.0 (#1168) Bumps [Swatinem/rust-cache](https://github.com/swatinem/rust-cache) from 2.6.2 to 2.7.0. - [Release notes](https://github.com/swatinem/rust-cache/releases) - [Changelog](https://github.com/Swatinem/rust-cache/blob/master/CHANGELOG.md) - [Commits](Swatinem/rust-cache@e207df5...a95ba19) --- updated-dependencies: - dependency-name: Swatinem/rust-cache dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump jsonrpsee from 0.20.0 to 0.20.1 (#1165) Bumps [jsonrpsee](https://github.com/paritytech/jsonrpsee) from 0.20.0 to 0.20.1. - [Release notes](https://github.com/paritytech/jsonrpsee/releases) - [Changelog](https://github.com/paritytech/jsonrpsee/blob/v0.20.1/CHANGELOG.md) - [Commits](paritytech/jsonrpsee@v0.20.0...v0.20.1) --- updated-dependencies: - dependency-name: jsonrpsee dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump clap from 4.4.2 to 4.4.3 (#1163) Bumps [clap](https://github.com/clap-rs/clap) from 4.4.2 to 4.4.3. - [Release notes](https://github.com/clap-rs/clap/releases) - [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md) - [Commits](clap-rs/clap@v4.4.2...v4.4.3) --- updated-dependencies: - dependency-name: clap dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix double quote import introduced by merge * clippy * put ensure_unique_types on metadata, revert toml changes * fmt * solve frame_metadata import * tidy fetch-metadata and non_exhaustive on errors * remove the getrandom thing * fix toml autoformatting --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: James Wilson <james@jsdw.me>
paritytech · Sep 20, 2023 · 12e3434 · 12e3434
1 parent e533cd3
commit 12e3434
Show file tree

Hide file tree

Showing 12 changed files with 271 additions and 164 deletions.
diff --git a/cli/src/commands/compatibility.rs b/cli/src/commands/compatibility.rs
@@ -77,7 +77,7 @@ async fn handle_pallet_metadata(
                 compatibility
                     .pallet_present
                     .entry(hex_hash)
-                    .or_insert_with(Vec::new)
+                    .or_default()
                     .push(node.to_string());
             }
             None => {
@@ -110,7 +110,7 @@ async fn handle_full_metadata(
 
         compatibility_map
             .entry(hex_hash)
-            .or_insert_with(Vec::new)
+            .or_default()
             .push(node.to_string());
     }
 

diff --git a/cli/src/commands/diff.rs b/cli/src/commands/diff.rs
@@ -13,7 +13,6 @@ use scale_info::Variant;
 
 use subxt_metadata::{
     ConstantMetadata, Metadata, PalletMetadata, RuntimeApiMetadata, StorageEntryMetadata,
-    StorageEntryType,
 };
 
 /// Explore the differences between two nodes
@@ -220,40 +219,34 @@ impl StorageEntryDiff {
         metadata_1: &Metadata,
         metadata_2: &Metadata,
     ) -> Self {
-        let value_1_ty_id = match storage_entry_1.entry_type() {
-            StorageEntryType::Plain(value_ty) | StorageEntryType::Map { value_ty, .. } => value_ty,
-        };
+        let value_1_ty_id = storage_entry_1.entry_type().value_ty();
         let value_1_hash = metadata_1
-            .type_hash(*value_1_ty_id)
+            .type_hash(value_1_ty_id)
             .expect("type should be present");
-        let value_2_ty_id = match storage_entry_2.entry_type() {
-            StorageEntryType::Plain(value_ty) | StorageEntryType::Map { value_ty, .. } => value_ty,
-        };
+        let value_2_ty_id = storage_entry_2.entry_type().value_ty();
         let value_2_hash = metadata_1
-            .type_hash(*value_2_ty_id)
+            .type_hash(value_2_ty_id)
             .expect("type should be present");
         let value_different = value_1_hash != value_2_hash;
 
-        let key_1_hash = match storage_entry_1.entry_type() {
-            StorageEntryType::Plain(_) => None,
-            StorageEntryType::Map { key_ty, .. } => Some(*key_ty),
-        }
-        .map(|key_ty| {
-            metadata_1
-                .type_hash(key_ty)
-                .expect("type should be present")
-        })
-        .unwrap_or_default();
-        let key_2_hash = match storage_entry_2.entry_type() {
-            StorageEntryType::Plain(_) => None,
-            StorageEntryType::Map { key_ty, .. } => Some(*key_ty),
-        }
-        .map(|key_ty| {
-            metadata_2
-                .type_hash(key_ty)
-                .expect("type should be present")
-        })
-        .unwrap_or_default();
+        let key_1_hash = storage_entry_1
+            .entry_type()
+            .key_ty()
+            .map(|key_ty| {
+                metadata_1
+                    .type_hash(key_ty)
+                    .expect("type should be present")
+            })
+            .unwrap_or_default();
+        let key_2_hash = storage_entry_2
+            .entry_type()
+            .key_ty()
+            .map(|key_ty| {
+                metadata_2
+                    .type_hash(key_ty)
+                    .expect("type should be present")
+            })
+            .unwrap_or_default();
         let key_different = key_1_hash != key_2_hash;
 
         StorageEntryDiff {

diff --git a/codegen/Cargo.toml b/codegen/Cargo.toml
@@ -12,21 +12,25 @@ documentation = "https://docs.rs/subxt-codegen"
 homepage.workspace = true
 description = "Generate an API for interacting with a substrate node from FRAME metadata"
 
+[features]
+default = ["fetch-metadata"]
+fetch-metadata = ["dep:jsonrpsee", "dep:tokio", "dep:frame-metadata"]
+
 [dependencies]
 codec = { package = "parity-scale-codec", workspace = true, features = ["derive"] }
-frame-metadata = { workspace = true }
+frame-metadata = { workspace = true, optional = true }
 heck = { workspace = true }
 proc-macro2 = { workspace = true }
 quote = { workspace = true }
 syn = { workspace = true }
 scale-info = { workspace = true }
 subxt-metadata = { workspace = true }
-jsonrpsee = { workspace = true, features = ["async-client", "client-ws-transport-native-tls", "http-client"] }
+jsonrpsee = { workspace = true, features = ["async-client", "client-ws-transport-native-tls", "http-client"], optional = true }
 hex = { workspace = true }
-tokio = { workspace = true, features = ["rt-multi-thread"] }
+tokio = { workspace = true, features = ["rt-multi-thread"], optional = true }
 thiserror = { workspace = true }
 
 [dev-dependencies]
 bitvec = { workspace = true }
 scale-info = { workspace = true, features = ["bit-vec"] }
-pretty_assertions = { workspace = true }
+pretty_assertions = { workspace = true }
diff --git a/codegen/src/api/mod.rs b/codegen/src/api/mod.rs
@@ -20,14 +20,14 @@ use crate::error::CodegenError;
 use crate::{
     ir,
     types::{CompositeDef, CompositeDefFields, TypeGenerator, TypeSubstitutes},
-    utils::{fetch_metadata_bytes_blocking, MetadataVersion, Url},
     CratePath,
 };
+
 use codec::Decode;
 use heck::ToSnakeCase as _;
 use proc_macro2::TokenStream as TokenStream2;
 use quote::{format_ident, quote};
-use std::{collections::HashMap, fs, io::Read, path, string::ToString};
+use std::{fs, io::Read, path, string::ToString};
 use syn::parse_quote;
 
 /// Generate the runtime API for interacting with a substrate runtime.
@@ -146,7 +146,10 @@ impl GenerateRuntimeApi {
     /// # Warning
     ///
     /// Not recommended to be used in production environments.
-    pub fn generate_from_url(self, url: Url) -> Result<TokenStream2, CodegenError> {
+    #[cfg(feature = "fetch-metadata")]
+    pub fn generate_from_url(self, url: crate::utils::Url) -> Result<TokenStream2, CodegenError> {
+        use crate::utils::{fetch_metadata_bytes_blocking, MetadataVersion, Url};
+
         fn fetch_metadata(url: Url, version: MetadataVersion) -> Result<Metadata, CodegenError> {
             let bytes = fetch_metadata_bytes_blocking(url, version)?;
             Ok(Metadata::decode(&mut &bytes[..])?)
@@ -225,49 +228,10 @@ impl RuntimeGenerator {
     ///
     /// Supported versions: v14 and v15.
     pub fn new(mut metadata: Metadata) -> Self {
-        Self::ensure_unique_type_paths(&mut metadata);
+        metadata.ensure_unique_type_paths();
         RuntimeGenerator { metadata }
     }
 
-    /// Ensure that every unique type we'll be generating or referring to also has a
-    /// unique path, so that types with matching paths don't end up overwriting each other
-    /// in the codegen. We ignore any types with generics; Subxt actually endeavours to
-    /// de-duplicate those into single types with a generic.
-    fn ensure_unique_type_paths(metadata: &mut Metadata) {
-        let mut visited_path_counts = HashMap::<Vec<String>, usize>::new();
-        for ty in metadata.types_mut().types.iter_mut() {
-            // Ignore types without a path (ie prelude types).
-            if ty.ty.path.namespace().is_empty() {
-                continue;
-            }
-
-            let has_valid_type_params = ty.ty.type_params.iter().any(|tp| tp.ty.is_some());
-
-            // Ignore types which have generic params that the type generation will use.
-            // Ordinarily we'd expect that any two types with identical paths must be parameterized
-            // in order to share the path. However scale-info doesn't understand all forms of generics
-            // properly I think (eg generics that have associated types that can differ), and so in
-            // those cases we need to fix the paths for Subxt to generate correct code.
-            if has_valid_type_params {
-                continue;
-            }
-
-            // Count how many times we've seen the same path already.
-            let visited_count = visited_path_counts
-                .entry(ty.ty.path.segments.clone())
-                .or_default();
-            *visited_count += 1;
-
-            // alter the type so that if it's been seen more than once, we append a number to
-            // its name to ensure that every unique type has a unique path, too.
-            if *visited_count > 1 {
-                if let Some(name) = ty.ty.path.segments.last_mut() {
-                    *name = format!("{name}{visited_count}");
-                }
-            }
-        }
-    }
-
     /// Generate the API for interacting with a Substrate runtime.
     ///
     /// # Arguments

diff --git a/codegen/src/api/storage.rs b/codegen/src/api/storage.rs
@@ -96,10 +96,7 @@ fn generate_storage_entry_fns(
     };
 
     let snake_case_name = storage_entry.name().to_snake_case();
-    let storage_entry_ty = match storage_entry.entry_type() {
-        StorageEntryType::Plain(ty) => *ty,
-        StorageEntryType::Map { value_ty, .. } => *value_ty,
-    };
+    let storage_entry_ty = storage_entry.entry_type().value_ty();
     let storage_entry_value_ty = type_gen.resolve_type_path(storage_entry_ty);
     let docs = storage_entry.docs();
     let docs = should_gen_docs

diff --git a/codegen/src/error.rs b/codegen/src/error.rs
@@ -8,6 +8,7 @@ use proc_macro2::{Span, TokenStream as TokenStream2};
 
 /// Error returned when the Codegen cannot generate the runtime API.
 #[derive(Debug, thiserror::Error)]
+#[non_exhaustive]
 pub enum CodegenError {
     /// The given metadata type could not be found.
     #[error("Could not find type with ID {0} in the type registry; please raise a support issue.")]
@@ -82,11 +83,13 @@ impl CodegenError {
 
 /// Error attempting to load metadata.
 #[derive(Debug, thiserror::Error)]
+#[non_exhaustive]
 pub enum FetchMetadataError {
     #[error("Cannot decode hex value: {0}")]
     DecodeError(#[from] hex::FromHexError),
     #[error("Cannot scale encode/decode value: {0}")]
     CodecError(#[from] codec::Error),
+    #[cfg(feature = "fetch-metadata")]
     #[error("Request error: {0}")]
     RequestError(#[from] jsonrpsee::core::Error),
     #[error("'{0}' not supported, supported URI schemes are http, https, ws or wss.")]
@@ -97,6 +100,7 @@ pub enum FetchMetadataError {
 
 /// Error attempting to do type substitution.
 #[derive(Debug, thiserror::Error)]
+#[non_exhaustive]
 pub enum TypeSubstitutionError {
     /// Substitute "to" type must be an absolute path.
     #[error("`substitute_type(with = <path>)` must be a path prefixed with 'crate::' or '::'")]

diff --git a/codegen/src/lib.rs b/codegen/src/lib.rs
@@ -50,10 +50,17 @@ mod error;
 mod ir;
 mod types;
 
+#[cfg(feature = "fetch-metadata")]
 pub mod utils;
 
+#[cfg(all(feature = "web", feature = "fetch-metadata"))]
+compile_error!("subxt-codegen: the features 'web' and 'fetch_metadata' cannot be used together.");
+
 pub use self::{
     api::{GenerateRuntimeApi, RuntimeGenerator},
     error::{CodegenError, TypeSubstitutionError},
-    types::{CratePath, Derives, DerivesRegistry, Module, TypeGenerator, TypeSubstitutes},
+    types::{
+        CratePath, Derives, DerivesRegistry, Module, TypeDefGen, TypeDefParameters, TypeGenerator,
+        TypeSubstitutes,
+    },
 };
diff --git a/codegen/src/types/mod.rs b/codegen/src/types/mod.rs
@@ -262,6 +262,11 @@ impl<'a> TypeGenerator<'a> {
         self.derives.default_derives()
     }
 
+    /// Returns the type registry.
+    pub fn types(&self) -> &PortableRegistry {
+        self.type_registry
+    }
+
     /// Returns the derives to be applied to a generated type.
     pub fn type_derives(&self, ty: &Type<PortableForm>) -> Result<Derives, CodegenError> {
         let joined_path = ty.path.segments.join("::");

diff --git a/codegen/src/types/type_def.rs b/codegen/src/types/type_def.rs
@@ -118,6 +118,11 @@ impl TypeDefGen {
             ty_docs,
         })
     }
+
+    /// are there unused type params?
+    pub fn has_unused_type_params(&self) -> bool {
+        self.type_params.has_unused_type_params()
+    }
 }
 
 impl quote::ToTokens for TypeDefGen {

diff --git a/codegen/src/types/type_def_params.rs b/codegen/src/types/type_def_params.rs
@@ -60,6 +60,11 @@ impl TypeDefParameters {
     pub fn params(&self) -> &[TypeParameter] {
         &self.params
     }
+
+    /// Returns true if there are any unused type params
+    pub fn has_unused_type_params(&self) -> bool {
+        !self.unused.is_empty()
+    }
 }
 
 impl quote::ToTokens for TypeDefParameters {