add encointer runtime #80

brenzi · 2023-10-27T10:01:17Z

In fulfillment of RFC22, this PR adds the encointer runtime.

replacing #17

notes

this runtime shall enable async backing at the conservative 12s block time ("Phase 1" in Santi's tutorial). in a subsequent step we shall set target block rate of 6s
We used the same upstream dependency versions like the other system chains on master
The PR includes the runtime's commit history for reference
strictly using published crates.io dependencies

* Switch to custom `BlockAnnounceData` Instead of sending a `SignedFullStatement` this switches to a new struct `BlockAnnounceData` that is being send alongside the block announcement. The signed full statement contains the candidate commitments, meaning it could be a full runtime upgrade that we send alongside a block announcement... To prevent this, we now only send the candidate receipt and the compact statement. * Update to latest polkadot

…ws#313) This changes the collator to print an error if the block that we can not find is the genesis block, instead of only logging this as a `debug` message. This should help people when they have registered the wrong genesis state on the relay chain.

* Complete telemetry for parachain & relaychain * Update Substrate & Polkadot

…olkadot-fellows#315) * Block announce validation should use the correct `Validation` result The error variant is just for internal errors and we need to return `Failure` always when the other node send us an invalid statement. * Update network/src/lib.rs Co-authored-by: Sergei Shulepov <sergei@parity.io> Co-authored-by: Sergei Shulepov <sergei@parity.io>

* MQC auth Update polkadot WIP * Update polkadot * Silly syntax errors * Fix typo * Leave some comments and docs * Apply suggestions from code review Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Introduce the MessageQueueChain structure * Move the HRMP channel relevance check below * Fix the `receive_hrmp_after_pause` test * ValidationData is passed by reference * Replace "to cumulus" with "to the collator" * Update the test so that they are same as in polkadot Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Fix HRMP Location * Add call to xcm handler

…adot-fellows#317) * Update polkadot * Migrate all uses of MQC heads to merkle proofs * Mass rename `relay_parent_storage_root` * Restore parachain-system tests * Update polkadot and libp2p swarm for testing * Collapse match into an if let Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Fix compilation error in test-service Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Move pallets to pallets folder and rename them * Move genesis file to service * Rename primitives to primitives-core * Delete cumulus-runtime * Move stuff to client folder and rename

This renames and moves the `SystemInherentData` into its own crate. The struct is now called `ParachainInherentData`. Besides moving the struct, this also moves the code for creating this struct into this crate.

* Update Substrate & Polkadot * Update chainspecs * Update again to fix test

* Move consensus to consensus-common crate * Move the parachain consensus out of the collator * Add first relay chain consensus stuff * Remove some warnings * Fix more stuff * Fix collator test * Change `ParachainConsensus` to take a mutable self * Make everything compile * Feedback

…kadot-fellows#342) Currently validation data is shared by using a well known key between the parachain system pallet and the validate block implementation. This pr changes this by passing the parachain system directly to the validate block implementation to make use of it. Besides that, we also store the validation params in some thread local variable to make it inspectable by parachain system. This moves the validation of validation data and validation params to the parachain system pallet directly, instead of having this hidden inside the validate block implementation. Fixes: paritytech/cumulus#217

* Update to latest Substrate and Polkadot * log::debug! * Add log dependecies to runtime * Comma * Fix tests

* Add a command to purge the relay chain only * WIP * Update rococo-parachains/src/cli.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Move cli stuff to its own crate * Copyright dates * Test not working for some reason... * WIP * Revert "WIP" This reverts commit f97cd63742c7df822e4a6e52a29db5e0f56b7bfa. * Fix test to use provided relay chain * Apply suggestions from code review Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Add hint about which database could not be purged Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

…s#355)

* Update Substrate & Polkadot * Remove unused code

* Support xcm local execution in xcm handler. * Add docs.

* Create builder for test nodes * Fix syncing issue * Adds missing file

…ot-fellows#365)

…ntime

ggwpez · 2024-01-09T13:26:40Z

Some of the CI migration errors imply that there are pallet migrations missing. Especially for the scheduler 🙈

Balances: On chain storage version StorageVersion(0) doesn't match current storage version StorageVersion(1).
PolkadotXcm: On chain storage version StorageVersion(0) doesn't match current storage version StorageVersion(1).
Scheduler: On chain storage version StorageVersion(0) doesn't match current storage version StorageVersion(4).

I think we should not ignore this.

brenzi · 2024-01-09T13:31:44Z

system-parachains/encointer/src/lib.rs

+/// Migrations to apply on runtime upgrade.
+pub type Migrations = (
+	// fixing the scheduler with a local migration is necessary because we have missed intermediate
+	// migrations. the safest migration is, therefore, to clear all storage and bump StorageVersion
+	migrations_fix::scheduler::v4::MigrateToV4<Runtime>,
+	// also here we're actually too late with applying the migration. however, the migration does
+	// work as-is.
+	pallet_xcm::migration::v1::VersionUncheckedMigrateToV1<Runtime>,
+	// balances are more tricky. We missed to do the migration to V1 and now we have inconsistent
+	// state which can't be decoded to V0, yet the StorageVersion is V0.
+	// the strategy is to: just pretend we're on V1
+	migrations_fix::balances::v1::BruteForceToV1<Runtime>,
+	// then reset to V0
+	pallet_balances::migration::ResetInactive<Runtime>,
+	//then apply the proper migration as we should have done earlier
+	pallet_balances::migration::MigrateToTrackInactive<Runtime, xcm_config::CheckingAccount>,
+);
+


the CI in this repo have brought to light that we missed to perform storage migrations in the past. My guess is that this has gone unnoticed previously because of bogus feature propoagtion for try-runtime which has now been fixed with zepter (we did use try-runtime before and had migrations in our own pallets, but these have really gone unnoticed!!)

With these local extra migrations, we should be able to recover without damage

Okay thanks. It is rather simple to run the try-runtime check locally, in case you need to debug it (github is a bit slow currently).

https://github.com/paritytech/try-runtime-cli example here

runtimes/.github/workflows/check-migrations.yml

Line 79 in 98872cb

--runtime ./target/production/wbuild/$PACKAGE_NAME/$RUNTIME_BLOB_NAME \

Looks good now.

yes and no. it runs through and succeeds, but it complains about a corrupt state for one key:

error: [2024-01-09T14:08:47Z ERROR frame_support::storage] old key failed to decode at [227, 143, 24, 82, 7, 73, 138, 187, 92, 33, 61, 15, 176, 89, 179, 216, 216, 55, 28, 116, 117, 23, 21, 29, 134, 63, 146, 111, 129, 151, 24, 223, 191, 178, 127, 30, 174, 240, 107, 185, 3, 0, 0, 0, 96, 15, 180, 143, 173, 7, 149, 151, 21, 112, 71, 148, 112, 212, 184, 115, 3, 1, 0] converting key to hex: 0xE38F185207498ABB5C213D0FB059B3D8D8371C747517151D863F926F819718DFBFB27F1EAEF06BB903000000600FB48FAD0795971570479470D4B873030100 looking up onchain raw storage value hex: 0x600a000000000000000003000000

hard to guess what key and what value that is but I'll investigate

Looks like something in PolkadotXcm::VersionNotifyTargets.

good spot. I can confirm that!

from the looks of it, not much bad can happen here.
Can we accept that risk? I would. AFAIU that storage gets written anyway when we send an XCM to Encointer, right?

I will ask in the XCM chat what this means since i have on clue.

thank you. if we need to take action and modify the migrations, I suggest to let that be a separate PR.

Cleanest way is to remove this key altogether, however this would mean that the relay chain would not be notified of any XCM version changes on Encointer. To remedy this, the relay chain would have to resend a SubscribeVersion instruction to Encointer.

…ntime

brenzi · 2024-01-10T15:23:36Z

@joepetrowski your remarks have been addressed. May I ask you to check and consider approving?
CI passed, so I guess we could merge if you agree

joepetrowski · 2024-01-10T19:33:45Z

This should have a CHANGELOG entry too (and being probably the first post-release commit, under a new [Unreleased] section).

brenzi · 2024-01-10T20:57:38Z

/merge

fellowship-merge-bot · 2024-01-10T20:58:09Z

Enabled auto-merge in Pull Request

Available commands

/merge: Enables auto-merge for Pull Request
/merge cancel: Cancels auto-merge for Pull Request
/merge help: Shows this menu

For more information see the documentation

bkchr and others added 30 commits January 26, 2021 03:32

Update the chainspecs (polkadot-fellows#311)

0d086ae

Complete telemetry for parachain & relaychain (polkadot-fellows#301)

6438d32

* Complete telemetry for parachain & relaychain * Update Substrate & Polkadot

Fix HRMP Sibling Location (polkadot-fellows#318)

b32ce07

* Fix HRMP Location * Add call to xcm handler

The great refactor (polkadot-fellows#325)

fbacfe7

* Move pallets to pallets folder and rename them * Move genesis file to service * Rename primitives to primitives-core * Delete cumulus-runtime * Move stuff to client folder and rename

Move parachain inherent data into its own crate (polkadot-fellows#326)

886a1e1

This renames and moves the `SystemInherentData` into its own crate. The struct is now called `ParachainInherentData`. Besides moving the struct, this also moves the code for creating this struct into this crate.

New genesis (polkadot-fellows#327)

ec08d11

* Update Substrate & Polkadot * Update chainspecs * Update again to fix test

Update Substrate & Polkadot (polkadot-fellows#330)

f04ce06

Update Substrate & Polkadot again (polkadot-fellows#332)

28d1d00

Minor fixes and clarifications (polkadot-fellows#333)

da79d42

Deposit XCM execution success and fail events. (polkadot-fellows#338)

d8126cd

Update to latest Substrate & Polkadot (polkadot-fellows#341)

fc8c690

Same procedure as every day (polkadot-fellows#345)

6b38755

XCM handler: make send origin configurable. (polkadot-fellows#346)

9535ee2

Update to latest Substrate and Polkadot (polkadot-fellows#348)

16bc09f

* Update to latest Substrate and Polkadot * log::debug! * Add log dependecies to runtime * Comma * Fix tests

Update Polkadot & Substrate (polkadot-fellows#354)

937cd76

Remove frame-executive that isn't required anymore (polkadot-fellow…

b50848e

…s#355)

Update XcmError on sending failure. (polkadot-fellows#358)

56f60a1

Update Substrate & Polkadot (polkadot-fellows#359)

c7d2446

* Update Substrate & Polkadot * Remove unused code

Support xcm local execution in xcm-handler. (polkadot-fellows#357)

cb5a244

* Support xcm local execution in xcm handler. * Add docs.

Adapt code to new sc-telemetry (polkadot-fellows#362)

a3d50f2

Fix block propagation between non-collator nodes (polkadot-fellows#363)

cb9d101

* Create builder for test nodes * Fix syncing issue * Adds missing file

Update to latest Substrate & Polkadot and also move cli crate (polkad…

00bc5cd

…ot-fellows#365)

brenzi added 10 commits January 8, 2024 17:52

lift patches and toml alignment with other crates

f3623ce

Merge remote-tracking branch 'upstream/main' into ab/add-encointer-ru…

99b0be9

…ntime

merge fix and crate name alignment

0f87f5c

fmt

2574399

fmt nightly

e66deb5

add encointer to runtime matrix for CI

b2834bf

fix clippy

44b2ae8

bump encointer pallets to fix runtime-benchmarks build

b8917c4

taplo

4e7971b

revert out-of-scope changes by zepter

4f9efb2

pifragile mentioned this pull request Jan 9, 2024

Encointer Runtime #17

Closed

brenzi added 3 commits January 9, 2024 06:55

comment outdated migrations. kept for convenience

b99c40a

fmt

4665259

fmt nightly :-(

52ce8f8

storage migration fixes

d4a0605

brenzi commented Jan 9, 2024

View reviewed changes

brenzi and others added 4 commits January 9, 2024 14:51

dummy commit to re-trigger CI

24c8e70

dummy commit to re-trigger CI

f0f0b27

Merge remote-tracking branch 'upstream/main' into ab/add-encointer-ru…

f159ff6

…ntime

Merge branch 'main' into ab/add-encointer-runtime

5c4b98c

joepetrowski approved these changes Jan 10, 2024

View reviewed changes

brenzi added 2 commits January 10, 2024 20:43

add changelog

ee4fd2c

typo

13b81b7

fellowship-merge-bot bot merged commit 6334588 into polkadot-fellows:main Jan 10, 2024
14 checks passed

ggwpez mentioned this pull request Feb 28, 2024

Enable idempotency check #164

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add encointer runtime #80

add encointer runtime #80

brenzi commented Oct 27, 2023 •

edited

Loading

ggwpez commented Jan 9, 2024 •

edited

Loading

brenzi Jan 9, 2024 •

edited

Loading

ggwpez Jan 9, 2024

ggwpez Jan 9, 2024

brenzi Jan 9, 2024

ggwpez Jan 9, 2024

brenzi Jan 10, 2024 •

edited

Loading

ggwpez Jan 10, 2024 •

edited

Loading

brenzi Jan 10, 2024

KiChjang Jan 11, 2024

brenzi commented Jan 10, 2024 •

edited

Loading

joepetrowski commented Jan 10, 2024

brenzi commented Jan 10, 2024

fellowship-merge-bot bot commented Jan 10, 2024

add encointer runtime #80

add encointer runtime #80

Conversation

brenzi commented Oct 27, 2023 • edited Loading

ggwpez commented Jan 9, 2024 • edited Loading

brenzi Jan 9, 2024 • edited Loading

Choose a reason for hiding this comment

ggwpez Jan 9, 2024

Choose a reason for hiding this comment

ggwpez Jan 9, 2024

Choose a reason for hiding this comment

brenzi Jan 9, 2024

Choose a reason for hiding this comment

ggwpez Jan 9, 2024

Choose a reason for hiding this comment

brenzi Jan 10, 2024 • edited Loading

Choose a reason for hiding this comment

ggwpez Jan 10, 2024 • edited Loading

Choose a reason for hiding this comment

brenzi Jan 10, 2024

Choose a reason for hiding this comment

KiChjang Jan 11, 2024

Choose a reason for hiding this comment

brenzi commented Jan 10, 2024 • edited Loading

joepetrowski commented Jan 10, 2024

brenzi commented Jan 10, 2024

fellowship-merge-bot bot commented Jan 10, 2024

brenzi commented Oct 27, 2023 •

edited

Loading

ggwpez commented Jan 9, 2024 •

edited

Loading

brenzi Jan 9, 2024 •

edited

Loading

brenzi Jan 10, 2024 •

edited

Loading

ggwpez Jan 10, 2024 •

edited

Loading

brenzi commented Jan 10, 2024 •

edited

Loading