diff --git a/specs/test_formats/operations/README.md b/specs/test_formats/operations/README.md index 32cf880b36..fd9db9f046 100644 --- a/specs/test_formats/operations/README.md +++ b/specs/test_formats/operations/README.md @@ -21,8 +21,8 @@ Operations: | *`operation-name`* | *`operation-object`* | *`input name`* | *`processing call`* | |-------------------------|----------------------|----------------------|--------------------------------------------------------| -| `attestation` | `Attestation` | `attestation` | `process_attestation(state, attestation)` | -| `attester_slashing` | `AttesterSlashing` | `attester_slashing` | `process_attester_slashing(state, attester_slashing)` | +| `attestation` | `Attestation` | `attestation` | `process_attestation(state, attestation)` | +| `attester_slashing` | `AttesterSlashing` | `attester_slashing` | `process_attester_slashing(state, attester_slashing)` | | `block_header` | `Block` | `block` | `process_block_header(state, block)` | | `deposit` | `Deposit` | `deposit` | `process_deposit(state, deposit)` | | `proposer_slashing` | `ProposerSlashing` | `proposer_slashing` | `process_proposer_slashing(state, proposer_slashing)` | diff --git a/specs/test_formats/shuffling/README.md b/specs/test_formats/shuffling/README.md index 25074742d5..7f841aa5f1 100644 --- a/specs/test_formats/shuffling/README.md +++ b/specs/test_formats/shuffling/README.md @@ -7,10 +7,10 @@ Clients may take different approaches to shuffling, for optimizing, and supporting advanced lookup behavior back in older history. For implementers, possible test runners implementing testing can include: -1) Just test permute-index, run it for each index `i` in `range(count)`, and check against expected `output[i]` (default spec implementation). -2) Test un-permute-index (the reverse lookup; implemented by running the shuffling rounds in reverse, from `round_count-1` to `0`). -3) Test the optimized complete shuffle, where all indices are shuffled at once; test output in one go. -4) Test complete shuffle in reverse (reverse rounds, same as #2). +1) Just test `get_shuffled_index`. +2) Test the reverse lookup; implemented by running the shuffling rounds in reverse, from `round_count-1` to `0`. +3) Test the optimized complete shuffle, where all indices are shuffled at once. +4) Test complete shuffling in reverse (reverse rounds, similar to #2). ## Test case format @@ -20,13 +20,19 @@ count: int shuffled: List[int] ``` -- The `bytes32` is encoded a string, hexadecimal encoding, prefixed with `0x`. +- The `bytes32` is encoded as a string, hexadecimal encoding, prefixed with `0x`. - Integers are validator indices. These are `uint64`, but realistically they are not as big. -The `count` specifies the validator registry size. One should compute the shuffling for indices `0, 1, 2, 3, ..., count (exclusive)`. -Seed is the raw shuffling seed, passed to permute-index (or optimized shuffling approach). +The `count` specifies the count of validators being shuffled (i.e. active validators during committee computation). -## Condition +One should test the shuffling for indices `0, 1, 2, 3, ..., count (exclusive)`. + +`shuffled` is a mapping from `i` to `get_shuffled_index(i, count, seed)`. +- `i` here is the index within committee-partitioned space. `i...i+N (excl.)` is used to get the validators for a committee of size `N`. +- `get_shuffled_index(i, count, seed) -> int` returns the index within the active-validators space. Pointing to the validator assigned to the committee corresponding to `i`. -The resulting list should match the expected output `shuffled` after shuffling the implied input, using the given `seed`. +`seed` is the raw shuffling seed, passed to shuffling function. + +## Condition +For the `get_shuffled_index` implementation (or list-wise equivalent): `get_shuffled_index(i, count, seed) == shuffled[i]` diff --git a/specs/test_formats/ssz_generic/README.md b/specs/test_formats/ssz_generic/README.md index da0898087d..b35ded264e 100644 --- a/specs/test_formats/ssz_generic/README.md +++ b/specs/test_formats/ssz_generic/README.md @@ -1,12 +1,12 @@ # SSZ, generic tests This set of test-suites provides general testing for SSZ: - to instantiate any container/list/vector/other type from binary data. +to instantiate any container/list/vector/other type from binary data. Since SSZ is in a development-phase, the full suite of features is not covered yet. Note that these tests are based on the older SSZ package. The tests are still relevant, but limited in scope: - more complex object encodings have changed since the original SSZ testing. +more complex object encodings have changed since the original SSZ testing. A minimal but useful series of tests covering `uint` encoding and decoding is provided. This is a direct port of the older SSZ `uint` tests (minus outdated test cases). @@ -16,5 +16,7 @@ Test format documentation can be found here: [uint test format](./uint.md). *Note*: The current Phase 0 spec does not use larger uints, and uses byte vectors (fixed length) instead to represent roots etc. The exact uint lengths to support may be redefined in the future. -Extension of the SSZ tests collection is planned, with an update to the new spec-maintained `minimal_ssz.py`; - see CI/testing issues for progress tracking. +## Recommendation + +For SSZ testing directly applicable to test-networks, refer to the [`ssz_static`](../ssz_static/README.md) tests, +which cover SSZ behavior for phase-0, for all container types. diff --git a/specs/test_formats/ssz_static/core.md b/specs/test_formats/ssz_static/core.md index f24a225b08..444f1ecd29 100644 --- a/specs/test_formats/ssz_static/core.md +++ b/specs/test_formats/ssz_static/core.md @@ -22,7 +22,8 @@ A test-runner can implement the following assertions: - Serialization: After parsing the `value`, SSZ-serialize it: the output should match `serialized` - Hash-tree-root: After parsing the `value`, Hash-tree-root it: the output should match `root` - Optionally also check signing-root, if present. -- Deserialization: SSZ-deserialize the `serialized` value, and see if it matches the parsed `value` +- Deserialization: SSZ-deserialize the `serialized` value, and see if it matches the parsed `value`. + Note that this only covers valid inputs, SSZ implementations should be hardened for production in a later stage. ## References diff --git a/test_generators/README.md b/test_generators/README.md index 95d7e70a8c..2f4503fa0d 100644 --- a/test_generators/README.md +++ b/test_generators/README.md @@ -4,8 +4,12 @@ This directory contains all the generators for YAML tests, consumed by Eth 2.0 c Any issues with the generators and/or generated tests should be filed in the repository that hosts the generator outputs, here: [ethereum/eth2.0-spec-tests](https://github.com/ethereum/eth2.0-spec-tests). -Whenever a release is made, the new tests are automatically built, and -[eth2TestGenBot](https://github.com/eth2TestGenBot) commits the changes to the test repository. +Test-vectors are generated and published by the release-publishers for every release. + +Spec-tests are unified with the executable spec: these tests are PyTest compatible, run in CI, but also enable special settings. +Settings include: + - BLS on/off: see `ethspec.utils.bls.py > .bls_active`, and BLS-switch test decorators in `eth2spec.test.context.py` + - "generator mode": output the relevant test data into an encoded copy (`test_my_fn(generator_mode=True)`, enabled by decorating `test_my_fn` with `ethspec.test.utils.spectest()`) ## How to run generators