Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add asCallsOnlyArg to reduce size of metadata #149

Merged
merged 16 commits into from
Dec 1, 2021
Merged
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ export function createSignedTx(
signature: `0x${string}`,
options: OptionsWithMeta
): string {
const { metadataRpc, registry } = options;
registry.setMetadata(createMetadata(registry, metadataRpc));
const { metadataRpc, registry, asCallsOnlyArg } = options;
registry.setMetadata(createMetadata(registry, metadataRpc, asCallsOnlyArg));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit/Question: Why the suffix "Arg"?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea being that asCallsOnly is the name of the getter for the metadata in polkadot-js so I wanted to make it have a separate name to reduce confusion.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation!


const extrinsic = registry.createType(
'Extrinsic',
Expand Down
4 changes: 2 additions & 2 deletions packages/txwrapper-core/src/core/decode/decodeSignedTx.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ export function decodeSignedTx(
signedTx: string,
options: OptionsWithMeta
): DecodedSignedTx {
const { metadataRpc, registry } = options;
const { metadataRpc, registry, asCallsOnlyArg } = options;

registry.setMetadata(createMetadata(registry, metadataRpc));
registry.setMetadata(createMetadata(registry, metadataRpc, asCallsOnlyArg));

const tx = registry.createType('Extrinsic', hexToU8a(signedTx), {
isSigned: true,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ export function decodeSigningPayload(
signingPayload: string,
options: OptionsWithMeta
): DecodedSigningPayload {
const { metadataRpc, registry } = options;
const { metadataRpc, registry, asCallsOnlyArg } = options;

registry.setMetadata(createMetadata(registry, metadataRpc));
registry.setMetadata(createMetadata(registry, metadataRpc, asCallsOnlyArg));

// We use `createTypeUnsafe` here because it allows us to specify `withoutLog: true`,
// which silences an internal error message from polkadot-js. This is helpful in `decode`
Expand Down
4 changes: 2 additions & 2 deletions packages/txwrapper-core/src/core/decode/decodeUnsignedTx.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ export function decodeUnsignedTx(
unsigned: UnsignedTransaction,
options: OptionsWithMeta
): DecodedUnsignedTx {
const { metadataRpc, registry } = options;
const { metadataRpc, registry, asCallsOnlyArg } = options;

registry.setMetadata(createMetadata(registry, metadataRpc));
registry.setMetadata(createMetadata(registry, metadataRpc, asCallsOnlyArg));

const methodCall = registry.createType('Call', unsigned.method);
const method = toTxMethod(registry, methodCall);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
import { Metadata } from '@polkadot/types';
import { MetadataVersioned } from '@polkadot/types/metadata/MetadataVersioned';

import { polkadotV9122MetadataHex } from '../../test-helpers/metadata/polkadotV9122MetadataHex';
import { polkadotRegistryV9122 } from '../../test-helpers/registries';
import { createMetadata, createMetadataUnmemoized } from './createMetadata';

describe('createMetadata', () => {
const unmemoizedMetadata: Metadata = createMetadataUnmemoized(
polkadotRegistryV9122,
polkadotV9122MetadataHex
);
const unmemoizedMetadataAsCalls: MetadataVersioned = createMetadataUnmemoized(
polkadotRegistryV9122,
polkadotV9122MetadataHex,
true
);
const memoizedMetadata: Metadata = createMetadata(
polkadotRegistryV9122,
polkadotV9122MetadataHex
);
const memoizedMetadataAsCalls: MetadataVersioned = createMetadata(
polkadotRegistryV9122,
polkadotV9122MetadataHex,
true
);

it('Metadata should decrease in byte size when `asCallsOnlyArg` is true with `createMetadataUnmemoized`', () => {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the curious the reason we are testing both the unmemoized and memoized createMetadata is because if the length option for memoizee is changed to less than 3 the function will break for the asCallsOnlyArg, but createMetadataUnmemoized will still work with asCallsOnlyArg. I wanted to isolate them both so we can make sure in the future if any changes are made to either it still works as is for both.

const metadataBuffer = Buffer.from(unmemoizedMetadata.toHex(), 'utf-8');
const metadataAsCallsBuffer = Buffer.from(
unmemoizedMetadataAsCalls.toHex(),
'utf-8'
);

expect(metadataAsCallsBuffer.byteLength).toBeGreaterThan(0);
expect(metadataBuffer.byteLength).toBeGreaterThan(
metadataAsCallsBuffer.byteLength
);
});

it('Metadata should decrease in byte size when `asCallsOnlyArg` is true with `createMetadata`', () => {
const metadataBuffer = Buffer.from(memoizedMetadata.toHex(), 'utf-8');
const metadataAsCallsBuffer = Buffer.from(
memoizedMetadataAsCalls.toHex(),
'utf-8'
);

expect(metadataAsCallsBuffer.byteLength).toBeGreaterThan(0);
expect(metadataBuffer.byteLength).toBeGreaterThan(
metadataAsCallsBuffer.byteLength
);
});
});
15 changes: 10 additions & 5 deletions packages/txwrapper-core/src/core/metadata/createMetadata.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { Metadata } from '@polkadot/types';
import { TypeRegistry } from '@polkadot/types';
import { MetadataVersioned } from '@polkadot/types/metadata/MetadataVersioned';
import memoizee from 'memoizee';

/**
Expand All @@ -10,12 +11,15 @@ import memoizee from 'memoizee';
* @ignore
* @param registry - The registry of the metadata.
* @param metadata - The metadata as hex string.
* @param asCallsOnlyArg - Option to decreases the metadata to calls only
*/
function createMetadataUnmemoized(
export function createMetadataUnmemoized(
registry: TypeRegistry,
metadataRpc: string
): Metadata {
return new Metadata(registry, metadataRpc);
metadataRpc: string,
asCallsOnlyArg = false
): Metadata | MetadataVersioned {
const metadata = new Metadata(registry, metadataRpc);
return asCallsOnlyArg ? metadata.asCallsOnly : metadata;
}

/**
Expand All @@ -26,7 +30,8 @@ function createMetadataUnmemoized(
* @ignore
* @param registry - The registry of the metadata.
* @param metadata - The metadata as hex string.
* @param asCallsOnlyArg - Option to decreases the metadata to calls only
*/
export const createMetadata = memoizee(createMetadataUnmemoized, {
length: 2,
length: 3,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

briefly reading over the docs (https://github.com/medikoo/memoizee#arguments-length), I wonder if this would work: "Dynamic length behavior can be forced by setting length to false, that means memoize will work with any number of arguments."

I am also a little confused how default args work in terms of length because of this note "Parameters predefined with default values (ES2015+ feature) are not reflected in function's length, therefore if you want to memoize them as well, you need to tweak length setting accordingly".

Either way though sounds like this works, so maybe not worth looking into

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works with both false, and length 3 🤔 . false being more flexible, and 3 being strict but with a default arg. I do like the the idea of being strict with 3 though.

});
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@ export interface GetRegistryBaseArgs {
* Used to set the correct metadata for the registry
*/
metadataRpc: string;
/**
* Used to reduce the metadata size by only having the calls
*/
asCallsOnlyArg?: boolean;
}

/**
Expand All @@ -26,10 +30,11 @@ export function getRegistryBase({
chainProperties,
specTypes,
metadataRpc,
asCallsOnlyArg,
}: GetRegistryBaseArgs): TypeRegistry {
const registry = new TypeRegistry();

const metadata = createMetadata(registry, metadataRpc);
const metadata = createMetadata(registry, metadataRpc, asCallsOnlyArg);

registry.register(specTypes);

Expand Down
27 changes: 26 additions & 1 deletion packages/txwrapper-core/src/core/method/defineMethod.spec.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
import {
balancesTransfer,
POLKADOT_25_TEST_OPTIONS,
POLKADOT_9122_TEST_OPTIONS,
POLKADOT_9122_TEST_OPTIONS_CALLS_ONLY,
TEST_BASE_TX_INFO,
} from '../../test-helpers/constants';
TEST_METHOD_ARGS,
} from '../../test-helpers/';
import { defineMethod } from './defineMethod';

describe('defineMethod', () => {
Expand Down Expand Up @@ -57,4 +61,25 @@ describe('defineMethod', () => {
expect(unsigned.address).toBe(TEST_BASE_TX_INFO.address);
expect(unsigned.blockNumber).toBe('0x0041a58e');
});

it('Should have a smaller unsigned tx size when using `asCallsOnlyArg` with V14 formatted metadata', () => {
// balancesTransfer is a test helper that uses defineMethod to construct a unsigned tx
const unsignedPayload = balancesTransfer(
TEST_METHOD_ARGS.balances.transfer,
TEST_BASE_TX_INFO,
POLKADOT_9122_TEST_OPTIONS
);

const unsignedPayloadCallsOnly = balancesTransfer(
TEST_METHOD_ARGS.balances.transfer,
TEST_BASE_TX_INFO,
POLKADOT_9122_TEST_OPTIONS_CALLS_ONLY
);

expect(
Buffer.from(JSON.stringify(unsignedPayloadCallsOnly), 'utf-8').length
).toBeLessThan(
Buffer.from(JSON.stringify(unsignedPayload), 'utf-8').length
);
});
});
6 changes: 3 additions & 3 deletions packages/txwrapper-core/src/core/method/defineMethod.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ export function defineMethod(
info: TxInfo,
options: OptionsWithMeta
): UnsignedTransaction {
const { metadataRpc, registry } = options;
registry.setMetadata(createMetadata(registry, metadataRpc));
const { metadataRpc, registry, asCallsOnlyArg } = options;
registry.setMetadata(createMetadata(registry, metadataRpc, asCallsOnlyArg));

const tx = createDecoratedTx(registry, metadataRpc);

Expand Down Expand Up @@ -77,7 +77,7 @@ export function defineMethod(
})
.toHex(),
genesisHash: info.genesisHash,
metadataRpc,
metadataRpc: registry.metadata.toHex(),
method,
nonce: registry.createType('Compact<Index>', info.nonce).toHex(),
signedExtensions: registry.signedExtensions,
Expand Down
18 changes: 18 additions & 0 deletions packages/txwrapper-core/src/test-helpers/constants.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { getRegistryPolkadot } from './getRegistryPolkadot';
import { polkadotV9122MetadataHex } from './metadata/polkadotV9122MetadataHex';
import metadataRpc from './staticV3-1-1';
import metadataRpcV29 from './staticV4-3-1';
export { metadataRpc };
Expand Down Expand Up @@ -55,6 +56,23 @@ export const POLKADOT_29_TEST_OPTIONS = {
registry: getRegistryPolkadot(29, metadataRpcV29),
};

/**
* Test options for runtime v9122
*/
export const POLKADOT_9122_TEST_OPTIONS = {
metadataRpc: polkadotV9122MetadataHex,
registry: getRegistryPolkadot(9122, polkadotV9122MetadataHex),
};

/**
* Test options for runtime v9122 with calls only metadata
*/
export const POLKADOT_9122_TEST_OPTIONS_CALLS_ONLY = {
metadataRpc: polkadotV9122MetadataHex,
registry: getRegistryPolkadot(9122, polkadotV9122MetadataHex),
asCallsOnlyArg: true,
};

/**
* Dummy arguments for all methods we're testing.
*/
Expand Down

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export * from './polkadotRegistry';
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
import { getRegistryPolkadot } from '../getRegistryPolkadot';
import { polkadotV9122MetadataHex } from '../metadata/polkadotV9122MetadataHex';

/**
* Polkadot v9122 TypeRegistry
*/
export const polkadotRegistryV9122 = getRegistryPolkadot(
9122,
polkadotV9122MetadataHex
);
4 changes: 4 additions & 0 deletions packages/txwrapper-core/src/types/method.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ export interface OptionsWithMeta extends Options {
* The metadata of the runtime.
*/
metadataRpc: string;
/**
* Used to reduce the metadata size by only having the calls
*/
asCallsOnlyArg?: boolean;
}

/**
Expand Down
4 changes: 4 additions & 0 deletions packages/txwrapper-core/src/types/registry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,8 @@ export interface GetRegistryOptsCore {
* Chain ss58format, token decimals, and token ID
*/
properties?: ChainProperties;
/**
* Used to reduce the metadata size by only having the calls
*/
asCallsOnlyArg?: boolean;
}
21 changes: 20 additions & 1 deletion packages/txwrapper-examples/src/polkadot.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,26 @@ async function main(): Promise<void> {
'state_getRuntimeVersion'
);

// Create Polkadot's type registry.
/**
* Create Polkadot's type registry.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@emostov This is what i wrote for the examples. Let me know if you think there should be more than this.

*
* When creating a type registry, it accepts a `asCallsOnlyArg` option which
* defaults to false. When true this will minimize the size of the metadata
* to only include the calls. This removes storage, events, etc. This will
* ultimately decrease the size of the unsigned transaction.
*
* Example:
*
* ```
* const registry = getRegistry({
* chainName: 'Polkadot',
* specName,
* specVersion,
* metadataRpc,
* asCallsOnlyArg: true,
* });
* ```
*/
const registry = getRegistry({
chainName: 'Polkadot',
specName,
Expand Down
22 changes: 21 additions & 1 deletion packages/txwrapper-examples/src/polkadotBatchAll.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,26 @@ async function main(): Promise<void> {
);

// Create Polkadot's type registry.
/**
* Create Polkadot's type registry.
*
* When creating a type registry, it accepts a `asCallsOnlyArg` option which
* defaults to false. When true this will minimize the size of the metadata
* to only include the calls. This removes storage, events, etc. This will
* ultimately decrease the size of the unsigned transaction.
*
* Example:
*
* ```
* const registry = getRegistry({
* chainName: 'Polkadot',
* specName,
* specVersion,
* metadataRpc,
* asCallsOnlyArg: true,
* });
* ```
*/
const registry = getRegistry({
chainName: 'Polkadot',
specName,
Expand All @@ -56,7 +76,7 @@ async function main(): Promise<void> {
// Metadata and type defintion registry used to create the calls
const optionsWithMeta = {
registry: registry,
metadataRpc: metadataRpc,
metadataRpc: registry.metadata.toHex(),
};

// Arguments for 12 balances transferKeepAlive
Expand Down
2 changes: 1 addition & 1 deletion packages/txwrapper-examples/src/util.ts
Original file line number Diff line number Diff line change
Expand Up @@ -67,5 +67,5 @@ export function signWith(
})
.sign(pair);

return signature;
return signature as unknown as `0x${string}`;
}
2 changes: 2 additions & 0 deletions packages/txwrapper-registry/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ export function getRegistry({
specVersion,
metadataRpc,
properties,
asCallsOnlyArg,
}: GetRegistryOpts): TypeRegistry {
// Polkadot, kusama, and westend have known types in the default polkadot-js registry. If we are
// dealing with another network, use the apps-config types to fill the registry.
Expand All @@ -83,5 +84,6 @@ export function getRegistry({
// `getSpecTypes` is used to extract the chain specific types from the registries `knownTypes`
specTypes: getSpecTypes(registry, chainName, specName, specVersion),
metadataRpc,
asCallsOnlyArg,
});
}