A new more complete test format for ETH 2.0 testing

[protolambda]

The format

See specs/test_formats/README.md file in PR, includes new testing glossary, configuration considerations, and general format.

Test type formats will be documented in markdown files in the specs/test_formats/ folder.

Upgrade path:

• First, we just need consensus on the general format.
• Update the CI/pyspec PR (Combine specs and test-generators #851), it includes a base test-generator to make life easy
• Upgrade the existing test generators, documenting them in the progress
• Introduce new test types

Test Types

The following test-types are proposed:

• Existing types
  • BLS
  • SSZ
  • Shuffling
• Next up
  • general state transition, (pre state, [block...]) -> post state
  • transactions (operations), one handler per type (pre state, tx) -> post state
  • epoch sub-transitions (not the ones that just process a batch)
• To be looked into
  • fork choice tests
  • "deltas": test computed changes in balances
  • suggestions welcome

How it affects existing test types

BLS

To be updated to move the "sub types" to separate test suites, each with their own handler. I.e. it would look like:

├── bls
│   ├── signing
│   │   └── sign_msg.yml
│   ├── verify_single
│   │   ├── invalid.yml
│   │   └── valid.yml
│   ├── verify_multiple
│   │   ├── invalid.yml
│   │   └── valid.yml
│   ├── aggregate_sigs
│   │   ├── single.yml
│   │   ├── mixed_invalid.yml
│   │   └── multiple.yml
│   ├── aggregate_pubkeys
│   │   ├── single.yml
│   │   ├── two.yml
│   │   ├── multiple.yml
│   │   └── duplicate.yml
│   └── ...
...

SSZ

The SSZ tests are compatible as is, and the base generator is based of of the design of this generator.
Easy to port over to use the new base generator.

Shuffling

Although the current testing is good, it sort of mixes validator activation with the shuffling algorithm.
We may decide to change it to just test shuffling based on a list of indices later down the road.

[protolambda]

Work in progess, suggestions welcome, aiming at fast-paced progress to get new state transition tests in

[hwwhww]

From your tweet:

With this, together with a "fork timeline" format, we can have a standardized way of initializing testing contexts, and test networks. 👀 Close to the standardized genesis format ETH 1.0 never got to have

Side note: eth 1.0 does have WIP standardized genesis format: ethereum/EIPs#1085

"Fork-timeline" looks like the chain configuration in client side, take eth1 client for example since they have forks already: geth, trinity, harmony. In eth2, it looks like we can define it in a list of Fork objects.

I wonder what's the use case of having forks/chain configurable in tests? It looks like setting up a short-lived testnet and running the script of given operations - but the thing we want to test could be done with some normal slot-to-slot state transition tests too and might be more deterministic.

[protolambda]

@hwwhww

Side note: eth 1.0 does have WIP standardized genesis format

Yes, the timeline is very similar to the "params" section of the genesis format proposed for eth 1. But with slots, because we can :)

The "genesis" and "accounts" sections are not so compatible with eth 2: genesis of eth 2 is based on contract events on some eth-1-like chain. And I assume this is the case for a testnet as well. Either based on a connection with a real net, or just some raw contract log event data to inject in an eth 2 genesis setup.
We're not quite there yet with cross-client testnets, so I prefer to just focus on the forks timeline, and ensure the standard has a way of telling tests what fork to use when running e.g. a state-transition test with some arbitrary slot number in it.
Also, since tests are just based on input data, and there's no need for genesis data in most tests, maybe we should just keep it lean. We can already define test-nets with just a combination of a fork-timeline and a pointer to a deposit contract on a testnet or something.

[protolambda]

In eth2, it looks like we can define it in a list of Fork objects

@hwwhww Interesting, but stating both previous and current version for every entry in the list seems very verbose. Like spelling out a raw linked list. It does bring up an interesting point however: do we want forks based on epoch number (less flexible, current Fork format) or slot number (very exact, old Fork format)

[hwwhww]

The "genesis" and "accounts" sections are not so compatible with eth 2: genesis of eth 2 is based on contract events on some eth-1-like chain. And I assume this is the case for a testnet as well. Either based on a connection with a real net, or just some raw contract log event data to inject in an eth 2 genesis setup.

Yes, agreed that eth2 genesis format will be quite different from eth1. Just FYI. 🙂

a way of telling tests what fork to use when running e.g. a state-transition test with some arbitrary slot number in it.

For the state transition test, my gut tells me that one fork setting (i.e., set that this state transition happens with which fork) is good enough for most cases. But not sure if there's any case of mulit-fork timeline requirement?

but stating both previous and current version for every entry in the list seems very verbose

Right, but state transition tests probably only need one Fork object!

do we want forks based on epoch number (less flexible, current Fork format) or slot number (very exact, old Fork format)

I'd say based on current phase 0 design, it makes more sense to do with epoch only - unless we do another new fundemental change in the future... 😢

---

After all, the two different angles of test format designs so far:

1. Generic: as this PR, one general format for all cases
2. Categories: the current test formats, different formats for different tests (e.g., bls, shuffling, ssz)

I do like this solution more! 👍

[djrtwo]

I'd say based on current phase 0 design, it makes more sense to do with epoch only - unless we do another new fundemental change in the future...

Agreed -- epoch number.

Also agree that using the Fork data structure to specify a chain is overkill. We just maintain current and prev version in the BeaconState to ensure we can process signatures from right before and after the fork. The fork timeline list will naturally fit into the Fork construct as needed.

But not sure if there's any case of mulit-fork timeline requirement?

Might be useful to test a fork in rapid succession (kind of like the constantinople and petersburg forks).

[protolambda]

Changed fork timeline definition to use epoch numbers

[hwwhww]

LGTM :)

[djrtwo]

a couple of nitpicks, then ready to merge