Write Test Runners for Spec Tests in Their Respective Packages

[rauljordan]

Hi all,

This is a tracking issue to ensure Prysm passes the latest Eth2 spec tests as maintained here. Every YAML file has a specific structure to it with specific expectations and configuration values. As discussed by the team offline, we agreed to keep these test runners constrained to the specific package they target. For example, the spec test runner for block processing should be kept in

/beacon-chain
  /core
    /blocks
      /spectest
        autogenerated.yaml.go
        block_processing_spec_test.go
      block_processing.go

This runner will have specific Go code for parsing and running the block sanity tests. This work can be split up, and shuffling spec tests have already been handled by @shayzluf here, but they should still be moved somewhere closer to the core shuffling logic in a respective /spectest folder as outlined above. The TODOS are as follow:

• Isolate shuffling related spec tests and their runner closer to the shuffling core logic Shuffle tests revisited #2829
• Implement block processing sanity tests runner Block Processing Sanity Spec Tests #2817
• Implement slot processing sanity tests runner Slot processing spec test #2813
• Implement epoch processing crosslink tests runner Epoch processing spec tests #2814
• Implement epoch processing registry updates runner Epoch processing spec tests #2814
• Implement block attestation operations test runner Block operations spec tests #2828
• Implement block attester slashing operations test runner Block operations spec tests #2828
• Implement block block header operations test runner Block operations spec tests #2828
• Implement block deposit operations test runner Block operations spec tests #2828
• Implement block proposer slashing operations test runner Block operations spec tests #2828
• Implement block transfer operations test runner Block operations spec tests #2828
• Implement block voluntary exit operations test runner Block operations spec tests #2828
• Implement SSZ generic/core test runner
• Implement BLS tests BLS spec tests #2826
• Implement finalization and justification tests Justification spec tests #2896
• Implement final updates tests Final updates spec tests #2901
• Genesis trigger spec tests Create Test Runner for Genesis State Spec Tests #2940
• Update YAMLs from spec freeze

[prestonvanloon]

Example gist of using upstream eth2 yaml as a runfile data via bazel: https://gist.github.com/prestonvanloon/7beb97de9d22651dd5bf318135bb4520

Please choose the appropriate test size https://docs.bazel.build/versions/master/test-encyclopedia.html#role-of-the-test-runner

Please tag all of the spec tests with spectest.

# Example test target (exclude the comments in prod BUILD files)
go_test(
   name = "go_default_test",
   ...
    
   size = "large", # Implied test timeout of 900s; choose appropriate size.
   tags = [
        "spectest", # So we can query for spec tests.
    ],
)

Edit: do not attempt to use bazel as the upstream dep manager for yaml files and generated go files yet.
Use this tool for now: prysmaticlabs/yaml-to-go#8
We'll add that tool as starlark rules at a later date and remove the generated stuff.

Edit: I removed suggestion for manual tag. These tests are not as big as I expected.

Edit: We can should check in xxx.yaml.go files but not check in the yaml itself. That is provided from the forked repository where we upload archives. It is in master WORKSPACE now here:

prysm/WORKSPACE

Line 133 in 9bd6147

name = "eth2_spec_tests",

[rauljordan]

I am taking up SSZ

[prestonvanloon]

Is this a complete list of all yamls upstream?

tests
├── bls
│   ├── aggregate_pubkeys
│   ├── aggregate_sigs
│   ├── msg_hash_g2_compressed
│   ├── msg_hash_g2_uncompressed
│   ├── priv_to_pub
│   └── sign_msg
├── epoch_processing
│   ├── crosslinks
│   └── registry_updates
├── operations
│   ├── attestation
│   ├── attester_slashing
│   ├── block_header
│   ├── deposit
│   ├── proposer_slashing
│   ├── transfer
│   └── voluntary_exit
├── sanity
│   ├── blocks
│   └── slots
├── shuffling
│   └── core
├── ssz_generic
│   └── uint
└── ssz_static
    └── core

Edit: was missing BLS.

[prestonvanloon]

Waiting for spec freeze to cut new (final) yaml files to resolve this.

[prestonvanloon]

There are some new tests: https://gist.github.com/prestonvanloon/ad815f6bd4c132018f16f53b436e2440

[rauljordan]

Completed in the spec-v0.6 branch

[prestonvanloon]

There are still two PRs that are open:

#2901 - Final updates spec test
#2905 - Genesis trigger update
TBD - Genesis trigger spec tests

[prestonvanloon]

Some details about genesis yaml test here: #2934

[0xKiwi]

Genesis Spec Tests complete, #2940 builds fine!

[rauljordan]

Closed by #2940