Test format docs

specs/test_formats/operations/README.md

@@ -21,8 +21,8 @@ Operations:
 
 | *`operation-name`*      | *`operation-object`* | *`input name`*       | *`processing call`*                                    |
 |-------------------------|----------------------|----------------------|--------------------------------------------------------|
-| `attestation`           | `Attestation`        | `attestation`        | `process_attestation(state, attestation)`                  |
-| `attester_slashing`     | `AttesterSlashing`   | `attester_slashing`  | `process_attester_slashing(state, attester_slashing)`            |
+| `attestation`           | `Attestation`        | `attestation`        | `process_attestation(state, attestation)`              |
+| `attester_slashing`     | `AttesterSlashing`   | `attester_slashing`  | `process_attester_slashing(state, attester_slashing)`  |
 | `block_header`          | `Block`              | `block`              | `process_block_header(state, block)`                   |
 | `deposit`               | `Deposit`            | `deposit`            | `process_deposit(state, deposit)`                      |
 | `proposer_slashing`     | `ProposerSlashing`   | `proposer_slashing`  | `process_proposer_slashing(state, proposer_slashing)`  |

specs/test_formats/shuffling/README.md

@@ -7,10 +7,10 @@ Clients may take different approaches to shuffling, for optimizing,
  and supporting advanced lookup behavior back in older history.
 
 For implementers, possible test runners implementing testing can include:
-1) Just test permute-index, run it for each index `i` in `range(count)`, and check against expected `output[i]` (default spec implementation).
-2) Test un-permute-index (the reverse lookup; implemented by running the shuffling rounds in reverse, from `round_count-1` to `0`).
-3) Test the optimized complete shuffle, where all indices are shuffled at once; test output in one go.
-4) Test complete shuffle in reverse (reverse rounds, same as #2).
+1) Just test `get_shuffled_index`.
+2) Test the reverse lookup; implemented by running the shuffling rounds in reverse, from `round_count-1` to `0`.
+3) Test the optimized complete shuffle, where all indices are shuffled at once.
+4) Test complete shuffling in reverse (reverse rounds, similar to #2).
 
 ## Test case format
 
@@ -20,13 +20,19 @@ count: int
 shuffled: List[int]
 ```
 
-- The `bytes32` is encoded a string, hexadecimal encoding, prefixed with `0x`.
+- The `bytes32` is encoded as a string, hexadecimal encoding, prefixed with `0x`.
 - Integers are validator indices. These are `uint64`, but realistically they are not as big.
 
-The `count` specifies the validator registry size. One should compute the shuffling for indices `0, 1, 2, 3, ..., count (exclusive)`.
-Seed is the raw shuffling seed, passed to permute-index (or optimized shuffling approach). 
+The `count` specifies the count of validators being shuffled (i.e. active validators during committee computation).
 
-## Condition
+One should test the shuffling for indices `0, 1, 2, 3, ..., count (exclusive)`.
+
+`shuffled` is a mapping from `i` to `get_shuffled_index(i, count, seed)`.
+- `i` here is the index within committee-partitioned space. `i...i+N (excl.)` is used to get the validators for a committee of size `N`.
+- `get_shuffled_index(i, count, seed) -> int` returns the index within the active-validators space. Pointing to the validator assigned to the committee corresponding to `i`.
 
-The resulting list should match the expected output `shuffled` after shuffling the implied input, using the given `seed`.
+`seed` is the raw shuffling seed, passed to shuffling function. 
+
+## Condition
 
+For the `get_shuffled_index` implementation (or list-wise equivalent): `get_shuffled_index(i, count, seed) == shuffled[i]`

specs/test_formats/ssz_generic/README.md

@@ -1,12 +1,12 @@
 # SSZ, generic tests
 
 This set of test-suites provides general testing for SSZ:
- to instantiate any container/list/vector/other type from binary data.
+to instantiate any container/list/vector/other type from binary data.
 
 Since SSZ is in a development-phase, the full suite of features is not covered yet.
 Note that these tests are based on the older SSZ package.
 The tests are still relevant, but limited in scope:
- more complex object encodings have changed since the original SSZ testing.
+more complex object encodings have changed since the original SSZ testing.
 
 A minimal but useful series of tests covering `uint` encoding and decoding is provided.
 This is a direct port of the older SSZ `uint` tests (minus outdated test cases).
@@ -16,5 +16,7 @@ Test format documentation can be found here: [uint test format](./uint.md).
 *Note*: The current Phase 0 spec does not use larger uints, and uses byte vectors (fixed length) instead to represent roots etc.
 The exact uint lengths to support may be redefined in the future.
 
-Extension of the SSZ tests collection is planned, with an update to the new spec-maintained `minimal_ssz.py`;
- see CI/testing issues for progress tracking.
+## Recommendation
+
+For SSZ testing directly applicable to test-networks, refer to the [`ssz_static`](../ssz_static/README.md) tests,
+which cover SSZ behavior for phase-0, for all container types.

specs/test_formats/ssz_static/core.md

@@ -22,7 +22,8 @@ A test-runner can implement the following assertions:
 - Serialization: After parsing the `value`, SSZ-serialize it: the output should match `serialized`
 - Hash-tree-root: After parsing the `value`, Hash-tree-root it: the output should match `root`
     - Optionally also check signing-root, if present.
-- Deserialization: SSZ-deserialize the `serialized` value, and see if it matches the parsed `value`
+- Deserialization: SSZ-deserialize the `serialized` value, and see if it matches the parsed `value`.
+  Note that this only covers valid inputs, SSZ implementations should be hardened for production in a later stage.
 
 ## References
 

test_generators/README.md

@@ -4,8 +4,12 @@ This directory contains all the generators for YAML tests, consumed by Eth 2.0 c
 
 Any issues with the generators and/or generated tests should be filed in the repository that hosts the generator outputs, here: [ethereum/eth2.0-spec-tests](https://github.com/ethereum/eth2.0-spec-tests).
 
-Whenever a release is made, the new tests are automatically built, and
-[eth2TestGenBot](https://github.com/eth2TestGenBot) commits the changes to the test repository.
+Test-vectors are generated and published by the release-publishers for every release.
+
+Spec-tests are unified with the executable spec: these tests are PyTest compatible, run in CI, but also enable special settings.
+Settings include:
+ - BLS on/off: see `ethspec.utils.bls.py > .bls_active`, and BLS-switch test decorators in `eth2spec.test.context.py`
+ - "generator mode": output the relevant test data into an encoded copy (`test_my_fn(generator_mode=True)`, enabled by decorating `test_my_fn` with `ethspec.test.utils.spectest()`)
 
 ## How to run generators