A new more complete test format for ETH 2.0 testing

specs/core/0_beacon-chain.md

@@ -178,6 +178,9 @@ Code snippets appearing in `this style` are to be interpreted as Python code.
 
 ## Constants
 
+Note: the default main-net values for the constants are included here for illustrative purposes.
+The different configurations for main-net, test-nets, and yaml-based testing can be found in the `configs/contants/` directory.
+
 ### Misc
 
 | Name | Value |

specs/test_formats/README.md

@@ -0,0 +1,170 @@
+# General test format
+
+This document defines the YAML format and structure used for ETH 2.0 testing.
+
+## ToC
+
+* [About](#about)
+* [Glossary](#glossary)
+* [Test format philosophy](#test-format-philosophy)
+* [Test Suite](#yaml-suite)
+* [Config](#config)
+* [Fork-timeline](#fork-timeline)
+* [Config sourcing](#config-sourcing)
+* [Test structure](#test-structure)
+
+## About
+
+Ethereum 2.0 uses YAML as the format for all cross client tests. This document describes at a high level the general format to which all test files should conform.
+
+The particular formats of specific types of tests (test suites) are defined in separate documents.
+
+## Glossary
+
+- `generator`: a program that outputs one or more `suite` files.
+  - A generator should only output one `type` of test.
+  - A generator is free to output multiple `suite` files, optionally with different `handler`s.
+- `type`: the specialization of one single `generator`.
+- `suite`: a YAML file with:
+  - a header: describes the `suite`, and defines what the `suite` is for
+  - a list of test cases
+- `runner`: where a generator is a *"producer"*, this is the *"consumer"**.
+  - A `runner` focuses on *only one* `type`, and each type has *only one* `runner`.
+- `handler`: a `runner` may be too limited sometimes, you may have a `suite` with a specific focus that requires a different format.
+  To facilitate this, you specify a `handler`: the runner can deal with the format by using the specified handler.
+  Using a `handler` in a `runner` is optional.
+- `case`: a test case, an entry in the `test_cases` list of a `suite`. A case can be anything in general, 
+  but its format should be well-defined in the documentation corresponding to the `type` (and `handler`).\
+  A test has the same exact configuration and fork context as the other entries in the `case` list of its `suite`.
+- `forks_timeline`: a fork timeline definition, a YAML file containing a key for each fork-name, and a slot number as value.
+
+## Test format philosophy
+
+### Config design
+
+After long discussion, the following types of configured constants were identified:
+- Never changing: genesis data
+- Changing, but reliant on old value: e.g. a slot time may change, but if you want to do the conversion 
+  `(genesis data, timestamp) -> slot number` you end up needing both constants.
+- Changing, but kept around during fork transition: finalization may take a while,
+  e.g. an executable has to deal with new deposits and old deposits at the same time. Another example may be economic constants.
+- Additional, back-wards compatible: new constants are introduced for later phases
+- Changing: there is a very small chance some constant may really be *replaced*. 
+  In this off-chance, it is likely better to include it as an additional variable,
+  and some clients may simply stop supporting the old one, if they do not want to sync from genesis.
+
+Based on these types of changes, we model the config as a list of key value pairs,
+ that only grows with every fork (they may change in development versions of forks however, git manages this).
+With this approach, configurations are backwards compatible (older clients ignore unknown variables), and easy to maintain.
+
+### Fork config design
+
+There are two types of fork-data:
+1) timeline: when does a fork take place?
+2) coverage: what forks are covered by a test?
+
+The first is neat to have as a separate form: we prevent duplication, and can run with different presets
+ (e.g. fork timeline for a minimal local test, for a public testnet, or for main-net)
+
+The second is still somewhat ambiguous: some tests may want cover multiple forks, and can do so in different ways:
+- run one test, transitioning from one to the other
+- run the same test for both
+- run a test for every transition from one fork to the other
+- more
+
+There is a common factor here however: the options are exclusive, and give a clear idea on what test suites need to be ran to cover testing for a specific fork.
+The way this list of forks is interpreted, is up to the test-runner:
+State-transition test suites may want to just declare forks that are being covered in the test suite,
+ whereas shuffling test suites may want to declare a list of forks to test the shuffling algorithm for individually. 
+
+### Test completeness
+
+Tests should be independent of any sync-data. If one wants to run a test, the input data should be available from the YAML.
+The aim is to provide clients with a well-defined scope of work to run a particular set of test-suites.
+
+- Clients that are complete are expected to contribute to testing, seeking for better resources to get conformance with the spec, and other clients.
+- Clients that are not complete in functionality can choose to ignore suites that use certain test-runners, or specific handlers of these test-runners.
+- Clients that are on older versions can test there work based on older releases of the generated tests, and catch up with newer releases when possible.
+
+## Test Suite
+
+```
+title: <required, string, short, one line> -- Display name for the test suite
+summary: <required, string, average, 1-3 lines> -- Summarizes the test suite
+forks_timeline: <required, string, reference to a fork definition file, without extension> -- Used to determine the forking timeline
+forks: <required, list of strings> -- Runner decides what to do: run for each fork, or run for all at once, each fork transition, etc.
+  - ... <required, string, first the phase name, then the spec version>
+config: <required, string, reference to a config file, without extension> -- Used to determine which set of constants to run (possibly compile time) with
+runner: <required, string, no spaces, python-like naming format> *MUST be consistent with folder structure*
+handler: <optional, string, no spaces, python-like naming format> *MUST be consistent with folder structure*
+
+test_cases: <list, values being maps defining a test case each>
+   ...
+
+```
+
+## Config
+
+A configuration is a separate YAML file.
+Separation of configuration and tests aims to:
+- prevent duplication of a minimal set of tests
+- make all tests easy to upgrade when a new config constant is introduced.
+- clearly define which constants to use
+- share-able between clients, for cross-client short or long lived test-nets
+- minimize the amounts of different constants permutations to compile as a client. \**
+
+\**: Some clients prefer compile-time constants and optimizations.
+They should compile for each configuration once, and run the corresponding tests per build target.
+
+## Fork-timeline
+
+A fork timeline is (preferably) loaded in as a configuration object into a client, as opposed to the constants configuration:
+ - we do not allocate or optimize any code based on slot numbers
+ - when we transition from one fork to the other, it is preferred to stay online.
+ - we may decide on a slot number for a fork based on external events (e.g. Eth1 log event),
+    a client should be able to activate a fork dynamically.
+
+Note that phases are considered to be "super forks",
+ i.e. they will just have a fork name, and be more heavy on changes.
+
+## Config sourcing
+
+The constants configurations are located in:
+
+```
+<specs repo root>/configs/constants/<config name>.yaml
+```
+
+And copied by CI for testing purposes to:
+
+```
+<tests repo root>/configs/constants/<config name>.yaml
+```
+
+
+The fork timelines are located in:
+
+```
+<specs repo root>/configs/fork_timelines/<timeline name>.yaml
+```
+
+And copied by CI for testing purposes to:
+
+```
+<tests repo root>/configs/fork_timelines/<timeline name>.yaml
+```
+
+## Test structure
+
+To prevent parsing of hundreds of different YAML files to test a specific test type, 
+ or even more specific, just a handler, tests should be structured in the following nested form: 
+
+```
+.                        <--- root of eth2.0 tests repository
+├── bls                  <--- collection of handler for a specific test-runner, example runner: "bls"
+│   ├── signing          <--- collection of test suites for a specific handler, example handler: "signing". If no handler, use a dummy folder "main"
+│   │   ├── sign_msg.yml <--- an entry list of test suites
+│   │   ...              <--- more suite files (optional)
+│   ...                  <--- more handlers
+...                      <--- more test types
+```