metadata: Use v15 internally

[lexnv]

This PR upgrades the v14 metadata to the latest v15.

• Subxt can work with V14 and V15
• When loading a V14 metadata, the metadata is converted to V15 with additional added fields empty
• Pointing the CLI to a V14 metadata will return a V15

This lays the ground for the runtime metadata API work when a new substrate release includes paritytech/substrate#13302, that will include:

• fetch V15 metadata using the new API
• generate runtime API for calling into the runtime of substrate

[lexnv]

bot rebase

[niklasad1]

yeah, that type should really be copy

[jsdw]

I'm happy for this in this PR, but ultimately wonder whether we want to make the version of metadata we use in Subxt opaque, eg have a struct like

pub struct RuntimeMetadata(RuntimeMetadataV15);

and then having the functions in this crate accept this as an argument.

The idea is that then, changes to the metadata (esp as V15 is unstable and liable to change a little for instance) will not be a breaking change to Subxt, just an internal detail.

If we did this, there is a question about whether users may want access to the metadata that Subxt is using though. (Subxt wraps it and ends up with a Metadata struct, and currently users can call certain functions on that (the ones we use in Subxt internally). I don't really like that interface being exposed (I'd rather Metadata was entirely opaque) but a compromise is that we allow access to the underlying RuntimeMetadataV14 or whatever.

So ultimately I think we could:

• Have RuntimeMetadata as above to hide away the concrete version we are working with
• Make all of the functions on Metadata (in subxt lib crate) pub(crate); that way we can have exactly the interfacce we need as Subxt evolves internally without worrying about breaking things for users (because they can't use that stuff
• Give users access to the underlying metadata by impl RuntimeMetadata { pun fn into_runtime_metadata_v14(self) -> RuntimeMetadataV14 } and eg impl Metadata { pub fn runtime_metadata() } (so that user can do metadata.runtime_metadata().clone().into_runtime_metadata_v14(); yes you'd need to clone but it'd be a one off need; maybe we can do better?)

Then, in the future we can add into_runtime_metadata_v15() etc and so on as a purely additive non breaking change and the actual metadata we use internally never matters to users (and can be eg unstable versions etc without breaking anything)

What do you reckon?

As I said at the top, this could be a followup PR if it makes sense to people! @ascjones I wonder what you reckon to this too?

[jsdw]

(this would mean adding a way to go from latest metadata to V14, V15 etc but going backwards should almost entirely mean just throwing away some information, and if it ever is harder than that then we can opt to break and remove an into_runtime_metadata_vxx() fn if we want)

[jsdw]

@cmichi I have this memory that you wanted some function or data added to the Metadata struct in Subxt a while ago (and also a memory of thinking I'd eventually forget that it was wanted since Subxt doesn't need it).

If we stopped exposing all methods on Metadata but instead gave you a way to get back the RuntimeMetadataV14 as described above, would that be sufficient? I'm quite eager to hide said APIs if I can, because they basically just exist for Subxt internal usage and that's crept out into the real world a bit too much, so I think the frame-metadata structs are a better thing to provide access to :)

[jsdw]

Just to follow up; Alex pointed out on our call this morning that a bunch of traits rely on being able to get information out of metadata (and the functions exposed on Metadata, while they could be refined, exist as more optimal ways to get at what we need).

so, while we could have a RuntimeMetadata new type to hide the underlying metadata version we use, we'd still need to provide enough information in the Metadata interface to get everything anybody might need for decoding/encoding things.

So this needs more thought, basically!

[jsdw]

with my suggestion of pub struct RuntimeMetadata(RuntimeMetadataV15) above, this could be a fn like impl From<RuntimeMetadataV14> for RuntimeMetadata { ... }.

In general, I think we should avoid panicking in this public function, so until the above sort of thing happens I'd prefer that this didn't panic and either took a specific RuntimeMetadataV14 or whatever or returned an Option<RuntimeMetadataV15> so we can panic where it's called instead rather than potentially forget about the panicking at the call site

[lexnv]

I've removed this after all since we were only using it in:

• codegen while extracting the V15 metadata
• metadata crate for unit tests

We could add it back later if need be when we have a better grasp of the metadata :D

[jsdw]

I'm assuming all of this is just copy & paste and nothing has changed logic wise?

[lexnv]

Initially, the logic was moved from lib.rs to validation.rs. However, Niklas suggested some valid improvement ideas to make the code look cleaner, without impacting the functionality:

• moved a few hardcoded constants to a dedicated const
• renamed the function that concatenates two hashes
• a few places where we could copy from slices
• improved some bits of documentation

[jsdw]

Looking good to me; just a couple of small things, and a larger comment which could be a followup PR (I'd be interested in a bit of feedback on whether it's a sane idea anyway!)