test runner #7665

seunlanlege · 2020-12-03T13:06:21Z

Substrate test runner is a new testing kit built specifically for testing runtime upgrades and migrations, It can be used with an existing chain data, to assert that a new runtime upgrade should behave as expected before it even goes live on-chain.

The goal of this project is to have an e2e testing suite that is run on CI whenever runtime code changes, allowing us to test runtime upgrades/migrations in a stress-free manner, previous attempts at this

In order to achieve this goal, pallets need to migrate to using tests written specifically for the test runner, eg see pallet-balances-e2e

These tests are generic and so they can be used to test runtime upgrades/migrations for any runtime 🎊

gnunicorn · 2020-12-03T17:10:17Z

Any particular reason why we include the test-runner here rather than just adding the external repo as a dev-dependency where necessary?

seunlanlege · 2020-12-03T17:12:18Z

Well the test runner depends on substrate master. If we add as an external dependency, there'd be two different versions of substrate in use.

bkchr · 2020-12-03T17:41:14Z

For what do we need this test runner in substrate?

seunlanlege · 2020-12-03T18:54:31Z

For what do we need this test runner in substrate?

This project was originally started with the idea of testing runtime upgrades but in order to get there, we first need a test suite for all pallets. We'd like for these tests to be run in CI as another build step for PR's that touches runtime code and so it doesn't make sense for the tests/runner to live elsewhere.

bkchr · 2020-12-03T18:59:31Z

Please be more specific.

What will these pallet tests do, how will this pallet tests be used later for doing runtime upgrade tests?

How should that work when these tests are written for the Substrate runtime and you want to test a polkadot runtime upgrade?

seunlanlege · 2020-12-03T19:31:01Z

Pallet tests should cover general use case of pallets.

Runtime upgrades can be done on a testnet that tracks substrate releases.

Post upgrades, we can run the test suite using the test runner and assert that no pallet functionality is broken.

For polkadot runtime upgrades, polkadot specific pallets would also need their own tests which will live in the polkadot repo.

bkchr · 2020-12-03T19:40:51Z

We already have tests for all pallets that test their functionality.

seunlanlege · 2020-12-03T19:48:49Z

Right, yes we do, but not in a way we can use for testing runtime upgrades?

kianenigma · 2020-12-17T12:11:58Z

Everything that I built in substrate-debug-kit also has the same issue where:

Ideally I want it to live inside of substrate because I want it to depend on substrate master.
Ideally (from another perspective) is should live outside of substrate because it is external tooling.

Initially my stance was to go through bit of hassle, but keep this stuff outside of substrate, as I have done before.

I am ambivalent here and will review this next week again.

gnunicorn

Still unclear of whether this should live here, nor whether this does demonstrate more utility compared to the tests we already have. Is there any way to provide an node-upgrade-case?

gnunicorn · 2021-01-11T12:47:05Z

bin/node/cli/Cargo.toml

@@ -81,7 +81,7 @@ pallet-indices = { version = "2.0.0", path = "../../../frame/indices" }
 pallet-timestamp = { version = "2.0.0", default-features = false, path = "../../../frame/timestamp" }
 pallet-contracts = { version = "2.0.0", path = "../../../frame/contracts" }
 frame-system = { version = "2.0.0", path = "../../../frame/system" }
-pallet-balances = { version = "2.0.0", path = "../../../frame/balances" }
+pallet-balances = { version = "2.0.0", path = "../../../frame/balances", features = ["tests"] }


I don't think these tests should live here, they are not related to the node-cli and don't test its behaviour. No harm putting them into a new bin/node/e2e-tests-crate, that isn't being published. Would keep things nice and tidy.

gnunicorn · 2021-01-11T12:49:18Z

frame/balances/Cargo.toml

+pallet-sudo = { version = "2.0.0", path = "../../frame/sudo", optional = true }
+sp-keyring = { version = "2.0.0", path = "../../primitives/keyring", optional = true }
+substrate-test-runner = { version = "0.8.0", path = "../../test-utils/substrate-test-runner", optional = true }
+num-traits = { version = "0.2.14", optional = true }


same here, Why not put them into a sup-crate frame/balances/e2e-tests (non-publish until we've solved the binding issues)? They don't need (and should not need) access to internal types, thus they should be fine depending on the crate but don't have to live in it.

one more reason to get rid of frame/balances/e2e-tests.

Putting the same example in node/bin/testing or sth like that would make much more sense.

generic tests initial impl initial impl merged with master

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

kianenigma · 2021-03-10T13:51:04Z

thanks for the comment, I just disagree with the first one:

I worry about bloating the boilerplate set up for the test runner, runtime, txpool logs are imo sensible defaults

it is already bloated, because we are configuring much more unrelated stuff like networking and telemetry there in the configs, and I think the underlying issue is this, not the request to add configurable log target.

To me, the correct course of action is to simplify the current one: Specially building Configuration. We probably don't need to allow the full one to be set by user. Could have a TestRunnerConfig that has only a subset of Configuration, and have a From<TestRunnerConfig> for Configuration do the job for you under the hood.

Then, we probably don't need to worry about adding one extra function that allows you do do something like pub fn set_log(&mut self, target: &str, level: log::Level.

6, 7, 8 have also been addressed

I don't see a companion yet, but ok :D

…te-test-runner

bin/node/test-runner-example/src/lib.rs

…te into substrate-test-runner

seunlanlege · 2021-03-24T09:31:44Z

@bkchr please take a look at this once more

.

bkchr · 2021-03-24T09:47:05Z

Removed my requested change

ordian · 2021-03-25T14:38:04Z

test-utils/test-runner/Cargo.toml

+parity-scale-codec = "1.3.1"
+env_logger = "0.7.1"
+log = "0.4.8"
+futures01 = { package = "futures", version = "0.1.29" }
+futures = { package = "futures", version = "0.3", features = ["compat"] }
+rand = "0.7"
+tokio = { version = "0.2", features = ["full"] }
+libp2p = "0.35.1"


some of these are unused

seunlanlege added the A0-please_review Pull request needs code review. label Dec 3, 2020

seunlanlege requested a review from gnunicorn December 3, 2020 13:06

paritytech-ci added the D2-breaksapi label Dec 3, 2020

gnunicorn reviewed Jan 11, 2021

View reviewed changes

seunlanlege requested review from andresilva, athei, kianenigma, mxinden, NikVolf, pepyakin, shawntabrizi, sorpaas and tomusdrw as code owners January 12, 2021 13:37

substrate-test-runner

69732ef

generic tests initial impl initial impl merged with master

seunlanlege force-pushed the substrate-test-runner branch from aa69638 to 69732ef Compare January 12, 2021 13:41

seunlanlege removed request for andresilva, shawntabrizi, pepyakin, athei, NikVolf and kianenigma January 12, 2021 13:42

Seun Lanlege and others added 2 commits March 10, 2021 11:47

Apply suggestions from code review

edbafae

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Update test-utils/test-runner/src/lib.rs

f7c2358

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

introduce NodeConfig

5d7043f

seunlanlege mentioned this pull request Mar 11, 2021

Introduce test runner biolerplate paritytech/polkadot#2609

Merged

Seun Lanlege added 2 commits March 11, 2021 12:47

remove tokio-compat

3a8ca65

Merge branch 'master' of github.com:paritytech/substrate into substra…

9b5ed29

…te-test-runner

gnunicorn approved these changes Mar 12, 2021

View reviewed changes

Seun Lanlege added 2 commits March 16, 2021 17:19

Merge branch 'master' of github.com:paritytech/substrate into substra…

a821a5a

…te-test-runner

ignore test runner test

b394936

bkchr previously requested changes Mar 19, 2021

View reviewed changes

bin/node/test-runner-example/src/lib.rs Outdated Show resolved Hide resolved

kianenigma reviewed Mar 19, 2021

View reviewed changes

bin/node/test-runner-example/src/lib.rs Outdated Show resolved Hide resolved

Seun Lanlege and others added 2 commits March 19, 2021 19:03

fix doc test

216e1b7

fix test_runner test

6101c74

shawntabrizi requested a review from bkchr March 21, 2021 02:59

shawntabrizi and others added 4 commits March 20, 2021 23:00

Merge branch 'master' into substrate-test-runner

744a0f8

fix test

a17dfb4

Merge branch 'substrate-test-runner' of github.com:paritytech/substra…

bf51eb4

…te into substrate-test-runner

use Pallet instead

e741136

seunlanlege added the D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit label Mar 22, 2021

Seun Lanlege added 2 commits March 22, 2021 19:39

use no_run

a9d5900

use ignore?

a417c85

github-actions bot removed the A7-needspolkadotpr label Mar 22, 2021

seunlanlege merged commit 189d079 into master Mar 24, 2021

seunlanlege deleted the substrate-test-runner branch March 24, 2021 10:28

ordian reviewed Mar 25, 2021

View reviewed changes

hirschenberger pushed a commit to hirschenberger/substrate that referenced this pull request Apr 14, 2021

test runner (paritytech#7665)

01fb1c6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test runner #7665

test runner #7665

seunlanlege commented Dec 3, 2020 •

edited

Loading

gnunicorn commented Dec 3, 2020

seunlanlege commented Dec 3, 2020 •

edited

Loading

bkchr commented Dec 3, 2020

seunlanlege commented Dec 3, 2020

bkchr commented Dec 3, 2020

seunlanlege commented Dec 3, 2020 •

edited

Loading

bkchr commented Dec 3, 2020

seunlanlege commented Dec 3, 2020

kianenigma commented Dec 17, 2020

gnunicorn left a comment

gnunicorn Jan 11, 2021

gnunicorn Jan 11, 2021

kianenigma Mar 3, 2021

kianenigma commented Mar 10, 2021 •

edited

Loading

seunlanlege commented Mar 24, 2021

bkchr commented Mar 24, 2021

ordian Mar 25, 2021

test runner #7665

test runner #7665

Conversation

seunlanlege commented Dec 3, 2020 • edited Loading

gnunicorn commented Dec 3, 2020

seunlanlege commented Dec 3, 2020 • edited Loading

bkchr commented Dec 3, 2020

seunlanlege commented Dec 3, 2020

bkchr commented Dec 3, 2020

seunlanlege commented Dec 3, 2020 • edited Loading

bkchr commented Dec 3, 2020

seunlanlege commented Dec 3, 2020

kianenigma commented Dec 17, 2020

gnunicorn left a comment

Choose a reason for hiding this comment

gnunicorn Jan 11, 2021

Choose a reason for hiding this comment

gnunicorn Jan 11, 2021

Choose a reason for hiding this comment

kianenigma Mar 3, 2021

Choose a reason for hiding this comment

kianenigma commented Mar 10, 2021 • edited Loading

seunlanlege commented Mar 24, 2021

bkchr commented Mar 24, 2021

ordian Mar 25, 2021

Choose a reason for hiding this comment

seunlanlege commented Dec 3, 2020 •

edited

Loading

seunlanlege commented Dec 3, 2020 •

edited

Loading

seunlanlege commented Dec 3, 2020 •

edited

Loading

kianenigma commented Mar 10, 2021 •

edited

Loading