codegen broken by BinaryHeap in metadata

[niklasad1]

The latest metadata in polkadot contains BinaryHeap which seems not be supported by subxt (this was added in scale_info v2.11)

Message:  Unknown prelude type 'BinaryHeap'
Location: path/.cargo/registry/src/index.crates.io-6f17d22bba15001f/scale-typegen-0.2.0/src/typegen/type_path.rs:210

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

Reproducible steps:

1. git clone https://github.com/paritytech/polkadot-sdk && cd polkadot-sdk && git checkout d0ebb850ed2cefeb3e4ef8b8e0a16eb7fb6b3f3e
2. cargo run -p polkadot --release
# In another terminal
3. subxt metadata --version latest -f bytes > artifacts/metadata.scale
4. subxt codegen --file artifacts/metadata.scale | rustfmt > code.rs

I think we have make another release of subxt-cli to bump scale-typegen to support the latest version of scale-info

[niklasad1]

This will be fixed by bumping scale-typegen to v0.2.1 in the subxt-cli

[niklasad1]

Released now

[claravanstaden]

I am having some trouble with this issue still (I think it may be the same issue). The codegen works after upgrading to 0.35.2 but using the generated code gives me errors related to the BinaryHeap:

error[E0277]: the trait bound `EnqueuedOrder: std::cmp::Ord` is not satisfied
   --> src/parachains/relaychain.rs:1:1415422
    |
1   | ... > { :: subxt :: storage :: address :: Address :: new_static ("OnDemandAssignmentProvider" , "FreeEntries" , () , [229u8 , 190u8 ,...
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `std::cmp::Ord` is not implemented for `EnqueuedOrder`
    |
    = help: the trait `DecodeWithMetadata` is implemented for `DecodedValueThunk`
    = note: required for `BinaryHeap<EnqueuedOrder>` to implement `IntoVisitor`
    = note: required for `BinaryHeap<EnqueuedOrder>` to implement `DecodeAsType`
    = note: required for `BinaryHeap<EnqueuedOrder>` to implement `DecodeWithMetadata`
note: required by a bound in `subxt::storage::Address::<Keys, ReturnTy, Fetchable, Defaultable, Iterable>::new_static`
   --> /Users/claravanstaden/IdeaProjects/snowbridge/.cargo/registry/src/index.crates.io-6f17d22bba15001f/subxt-0.35.2/src/storage/storage_address.rs:95:15
    |
95  |     ReturnTy: DecodeWithMetadata,
    |               ^^^^^^^^^^^^^^^^^^ required by this bound in `Address::<Keys, ReturnTy, Fetchable, Defaultable, Iterable>::new_static`
...
100 |     pub fn new_static(
    |            ---------- required by a bound in this associated function

relaychain.rs is generated with: subxt codegen --url ws://localhost:9944 >src/parachains/relaychain.rs
Running against the Rococo config on https://github.com/paritytech/polkadot-sdk master branch. Please let me know what you think the issue may be. Perhaps it is a problem in my upgrade from subxt 0.33.0 to 0.35.2, but since the error originates from the generated code I think it may be a bug.

[niklasad1]

Can you share how you are generating your codegen i.e, are you deriving Ord for the all the generated types?

Perhaps you can probably workaround that by substituting the BinaryHeap type generated by subxt

#[subxt::subxt(
	runtime_metadata_path = "artifacts/metadata.scale",
	derive_for_all_types = "Clone, Debug, Eq, PartialEq",
        derive_for_type(
		path = "some_pallet::EnqueuedOrder",
		derive = "Ord"
	),
	substitute_type(
		path = "some_pallet:BinaryHeap",
		with = "::std::collections::BinaryHeap"
	),
)]
pub mod runtime {}

[jsdw]

We may need to map BinaryHeaps to Vec like we do for BTreeSet.

See

subxt/codegen/src/lib.rs

Line 389 in c07ee0c

parse_quote!(BTreeMap),

for the type substitution we do for BTreeSet (I think it may be as simple as adding a BinaryHeap thing in the same place here).

Why? Because BinaryHeap's codec impl requires Ord on the key type (as does most of its useful functions), but Ord isn't automatically derived on generated types, and so a BinaryHeap<Foo> where Foo is a generated type would probably lead to an error like the one described above.

So yeah; generating the code will work now because BinaryHeap<K> doesn't impose any restrictions itself on K, but using it is still an issue!

[jsdw]

@claravanstaden is there any chance you could try out this branch of Subxt?:

#1523

Fingers crossed that will resolve your issue, and if so, we'll merge it :)

[jsdw]

^ We tested it internally and it seemed to fix the issue, so for now this is closed but please do let us know if you run into something still with the latest master of Subxt

[niklasad1]

We have now released subxt v0.35.3 which fixes this issue, let us know if you still encounter any issues

[claravanstaden]

@niklasad1 @jsdw v0.35.3 fixed the issue for me, much appreciated!