Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

XCM v3: Bridge infrastructure #4186

Closed
wants to merge 101 commits into from
Closed

Conversation

gavofyork
Copy link
Member

@gavofyork gavofyork commented Oct 30, 2021

XCM protocol elements needed for bridging.

Introduces a conceptual Universal Ancestor under which all global consensus systems exist. Ancestry of a consensus system (chain) should now go back to this Universal Ancestor. A new junction type is introduced meant for global consensus systems which sit directly under the Universal Ancestor. NetworkId is reworked to remove the Any which allows it to be used within the new Junction::GlobalConsensus.

Local Consensus is defined as all locations under the same Global Consensus system as the local location. Non-Local Consensus is defined as all other locations. Bridges exist as a means of communication between systems which view each other as being under Non-Local Consensus.

Usage

There should be a standard router (NonLocalConsensus) for bridge-routing. Chains routers will include this in their router tuple. This will be configured with the XCM location(s) of the Bridge hub(s). This router will check for non-local consensus destinations (i.e. those which ascend to the Universal Ancestor and descend into a consensus which is not local). If a message is sent to such a destination and it is supported by a bridge, it will lookup the correct XCM location for the bridge then encompass the message in a Forward instruction, providing payment with a prefixed WithdrawAsset/BuyExecution if needed.

When a message is routed into a bridge, the bridge will check the origin and if a descendent of the overall consensus (which it should be representing), then a DescendOrigin will be prefixed. This message will then be queued for sending over the bridge together with the destination location relative to the destination consensus which the remote side of the bridge represents.

Bridge hubs will be configured as UniversalAliases for all networks that they bridge over to. When a message comes over a bridge they will send the message on to the destination consensus chain using intra-consensus transport, prefixed with a UniversalOrigin instruction with the source consensus system.

  • Rework NetworkId into Option<NetworkId> and remove NetworkId::Any.
  • New junction: Junction::GlobalConsensus(NetworkId)
  • New instruction: Instruction::UniversalOrigin
  • New xcm_executor::Config item: type UniversalAliases

CC @tomusdrw

TODO

  • New instruction: ExportMessage + xcm_executor::Config type item to configure it.
  • New SendXcm impl which detects destinations beyond our consensus and extracts the remote network and location within it. If a known (configurable by trait parameters) bridge exists for the remote network it prepends and packs the message and sends to the bridge, e.g.: [WithdrawAsset, BuyExecution, ExportMessage { network: remote_network, destination: remote_destination, xcm: Xcm(vec![UniversalOrigin(local_network), DescendOrigin(local_origin), { msg }]) }]. The type which configures this should be the same type which configures the UniversalAliases type.
  • Tests

GopherJ and others added 5 commits October 30, 2021 17:42
* increase ump_service_total_weight's default value to be the same with kusama

* remove comment
* collator-protocol/validator: do not wait 1s to poll requested collations

* collator-protocol/validator: do not request collation for the next group

* put everything into select

* fmt

* more hacks yay

* a test

* review nits

* remove outdated comment
@github-actions github-actions bot added the A0-please_review Pull request needs code review. label Oct 30, 2021
@gavofyork gavofyork changed the title XCM bridge infrastructure XCM v3: Bridge infrastructure Oct 31, 2021
@gavofyork gavofyork added B7-runtimenoteworthy C1-low PR touches the given topic and has a low impact on builders. D2-notlive 💤 PR contains changes in a runtime directory that is not deployed to a chain that requires an audit. labels Oct 31, 2021
gavofyork and others added 5 commits October 31, 2021 17:41
* bump rococo spec_version

* replace the spec file

* bump rococo chain id

* use rococo-staging
xcm/src/v3/mod.rs Outdated Show resolved Hide resolved
xcm/src/v3/mod.rs Outdated Show resolved Hide resolved
gavofyork and others added 5 commits November 2, 2021 17:12
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Bumps [dlmalloc](https://github.com/alexcrichton/dlmalloc-rs) from 0.2.1 to 0.2.2.
- [Release notes](https://github.com/alexcrichton/dlmalloc-rs/releases)
- [Commits](alexcrichton/dlmalloc-rs@0.2.1...0.2.2)

---
updated-dependencies:
- dependency-name: dlmalloc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.105 to 0.2.106.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](rust-lang/libc@0.2.105...0.2.106)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
emostov and others added 21 commits November 19, 2021 17:23
* chore/provisioner: move metrics to a separate module

* avoid the duplicate names

* reduce all checks

* fixup tests

* Update node/core/provisioner/src/lib.rs

Co-authored-by: Zeke Mostov <z.mostov@gmail.com>

* chore: fmt

* chore: spellcheck

* doc

* remove the enum anti-pattern

* guide update - remove all the responsibilities

* add another trivial check

* Update node/core/provisioner/src/metrics.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Update roadmap/implementers-guide/src/node/utility/provisioner.md

Co-authored-by: Andronik Ordian <write@reusable.software>

* Update node/core/provisioner/src/metrics.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

Co-authored-by: Zeke Mostov <z.mostov@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
* impl prefered items

Closes #4330

* do not stop attempting to select, just because one did not fit

* doc

* prefered -> preferred

* missing usage of the preferred indices

* sigh

* shuffle is not available for chacha

* remove duplicate weight addition

* ref vs no ref
* Companion for Taskmanager: Remove `clean_shutdown`

paritytech/substrate#10314

* Update Substrate

* Remove warning
* test/malus: craft the first maliciously disputing actor

* initial draft

* Add Dockerfile and instructions how to use it to build malus image locally

* Forgot one flag for the build cmd

* we are not docker specific, we are happy to use any containerruntime

* shuffle things around

* add initial tera based integration test

* chores

* fixins

* simple setup to start

* other samples (WIP)

* add Docker version with cargo-chef

* update substarte and small change of orders in commands in the container file

* metrics one

* fmt

* minor

* fixin

* fix metric names

* -d

* add open gauge

* fmt

* spellcheck

* fix test

* adjust to changed error messages

* refactor, more malus impls

* more malus changes

* foo

* minor cleanup

* suggest garbage candidate

* chore

* fix suggest garabge malus

* malus: back garbage candidate

* cargo lock

* re-introduce metrics

* chore: cargo fmt

* undoe 1.54.0 output, CI uses 1.53.0 rustc

* update location of js types

* Fix trybuild

* add tag to image name also; this will be replaced in the prod version

* Tests fixed

* add some fix me

* add dockerfile for ci

* Add docker file for malus for ci

* use variables in .toml file

* add chnages for malus test

* some fixes

* some more fixes

* Update .gitlab-ci.yml

* add local build for polkadot and malus

* some fixes

* enable disputes feature in CI

* ok, ok

* rename: MsgFilter -> MessageInterceptor

* remove TODO that would not have worked

* intercept

* refactor

* fix README and containers

* fix

* chore: cargo fmt

* avoid some more malus-$VARIANT references

* fix argument order

* chore: add about

* Update sanity check in relay chain selection

* fix order, add dispute-unavailable-block malus

* fixup: avoid underflow issue

* it's all u32

* fix conditional use

* Revert "it's all u32"

This reverts commit 6b3ae25.

* Revert "fixup: avoid underflow issue"

This reverts commit 336cbe2.

* Revert "Update sanity check in relay chain selection"

This reverts commit 970647f.

* update the malus bin

* Update node/malus/integrationtests/0003-dispute-unavailable-block.feature

Co-authored-by: Andronik Ordian <write@reusable.software>

* add some FIXME reminders

* update path to index.js

* Update .gitlab-ci.yml

* Update node/malus/integrationtests/0001-dispute-valid-block.toml

* try 1: make malus test run

* chore: cargo fmt

* temporary fix

* use subcommand syntax from latest gurke

* cargo +nightly fmt

* add collator to a a test

* docs: add env vars to README

* update ci to run dispute-valid-block test

* needs the polkadot image

* Fix path for nodejs container

* post merge fix

* download proper dir  containg configs for malus test

* update the malus ci job

* rm a whitespace

* temp build for malus

* use correct build command for temp malus

* remove subcommands for now

* set max validators per core in the default HostConfig

* tabs

* update beefy

* fixup

* fixup II

* make one variant compile

* make other variants compile

* revert changes to chain_spec

* fmt

* build malus image from polkadot-test-malus again

* revert unrelated changes

* try fixing build-malus job

* Revert "remove subcommands for now"

This reverts commit 5d8292b.

* try fixing build-malus job II

* MVP working dispute-ancestor

* renames

* fix PVF execution on malus

* fix test

* fix typo

* fmt

* checkmate

* try something

* make it actually work

* some tweaks to 01 feature test

* fmt

* sleep a bit more

* complete wococoization

* some tweaks to 01 feature test

* typo fix

* use correct metric names

* fix

* ffs

* .

* try some rearrangement

* Attempt to wait till initial node bootstrap in test

* Fix test syntax

* Run malus tests with v2 script

* Proper symlink created

* simnet v14

* add zombienet tests

* add zombie net test - draft

* add more tests to dispute suite

* add within to fix tests

* replace test directory and start test migration

* migrate all the tests

* add timeout to tests

* reduce debug

* make easy to test in dev

* set appropriated debug

* use image from ci

* fix config for test

* set images from ci

* fix config

* add COLIMAGE env

* tweek tests

* reduce debug

* typo

* wip, migrate old test to zombie-net

* adjunt test config for zombie-net

* run mauls 0001 test only

* clean and setup smoke-test in zombie-net

* add extra time to assertinons

* clean code to merge and improve README

* add info to access logs

* improved readme

* merge master and resolve conflicts

* Update zombienet_tests/README.md

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* clean and consolidate zombienet name

* change runner in gitlab

* add comment explain why we use wococo

* change tag for runner

* remove unused tests

* remove dup Dockerfile and update description

* fmt

* fix compilation post-merge

* fmt

* cut me Some slack

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: radupopa2010 <radupopa2010@yahoo.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Anton Gavrilov <AntonE.Gavrilov@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com>
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.107 to 0.2.108.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](rust-lang/libc@0.2.107...0.2.108)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* xcm added

* xcm pallet & config added Polkadot

* set advertise XCM version to 2 for Polkadot

* remove ChildSystemParachainAsSuperuser from Polkadot XCM config

* remove unpaid execution for system parachains

* remove unused imports

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>
* Add feature more feature gating for benchmarking + tests

* New line

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras_inherent.rs

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras_inherent.rs

* Do not assume we use max validators per core

* Use kusama weights for rococo (hopefully temp)

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras_inherent.rs

* Add more validity votes when neccesary

* Some fixes for the last commit

* Restore westend weights

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras_inherent.rs

* Revert bad westend weights write

* Make sure to update val idx before skipping

* Fix validity vote range to max at group size'

* Temp setup for rococo

* cargo run --quiet --release --features runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./runtime/rococo/src/weights/runtime_parachains_paras_inherent.rs --header=./file_header.txt

* Augment generated Rococo weights

* Make it compile

* Revert range for enter_backed_candidates_variable

* Delete runtime/kusama/src/weights/runtime_paras_paras_inherent.rs

Co-authored-by: Parity Bot <admin@parity.io>
* Metrics for InherentDataProvider

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Integrate metrics

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* more changes

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fmt

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* avoid naming confusion

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Move to Provisioner.

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* Add metric documentation

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
* minor: move checks into separate fn

* add additional validity checks

* simplify shuffling

* Closes potential OOB weight

* improve docs

* fooo

* remove obsolete comment

* move filtering into the rollback-transaction

Technically this is not necessary but avoids future footguns.

* move check up and avoid duplicate checks

* refactor: make sure backed candidates are sane, even more

* doc wording

Co-authored-by: Zeke Mostov <z.mostov@gmail.com>

* refactor: avoid const generics for sake of wasm size

`true` -> `FullCheck::Skip`, `false` -> `FullCheck::Yes`.

* chore: unify `CandidateCheckContext` instance names

* refactor: introduce `IndexedRetain` for `Vec<T>`

* chore: make tests prefix free

* doc: re-introduce removed comment

* refactor: remove another const generic to save some wasm size

Co-authored-by: Zeke Mostov <z.mostov@gmail.com>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
* add disputes to Kusama runtime

* Update DisputesHandler

* fmt

* add the benchmarks

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::disputes --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_disputes.rs

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Parity Bot <admin@parity.io>
* [ci] Added ssh token for publishing gh-pages

* returned github_token

* fix ssh commands for publish-rustdoc

* fix ssh command for publish-docs

* Fix publish-rustdoc, remove clear /tmp
* add Statemine as parachain 1000 to rococo trusted teleporters

* remove Rockmine/parachain 1001 from Rococo runtime
@gavofyork
Copy link
Member Author

Closed in favour of #4681 after accidentally merging master.

@gavofyork gavofyork deleted the gav-xcm-universal-origin branch January 10, 2022 13:53
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A0-please_review Pull request needs code review. C1-low PR touches the given topic and has a low impact on builders. D2-notlive 💤 PR contains changes in a runtime directory that is not deployed to a chain that requires an audit.
Projects
None yet
Development

Successfully merging this pull request may close these issues.