Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement proposed test organization and structure improvements #1320

Merged
merged 37 commits into from
Aug 3, 2019

Conversation

protolambda
Copy link
Contributor

@protolambda protolambda commented Jul 25, 2019

This PR implements the proposed changes to test vector generation in #1311

Change log:

  • Tests are structured as tests/<config name>/<fork or phase name>/<test runner name>/<test handler name>/<test suite name>/<test case>/<output part>. Background details can be found in Test orientation & distribution format issues #1311, and all the necessary information can be found in the new updated specs/test_formats/README.md
  • This effectively reduces resource requirements from running tests to the size of a single test case. This alleviates testing problems presented by Chainsafe and Trinity for the mainnet configuration.
  • For state-transition tests, all data is now also available in SSZ serialized form, as raw bytes in a .ssz file.
  • Test formats have generally not changed, the essential difference is that tests are split into multiple files, down to the pre/post/change level. This enables clients to choose for SSZ encoding over YAML. For better testing speed, and no issues with YAML unmarshalling. This alleviates pains Prysmatic Labs was having with the legacy choice for YAML.
  • A new experimental ssz_generic test generator is included, which utilizes the new ability for differently typed test parts, and starts to fill the previous gap of general-purposes SSZ tests.
  • Test generators can be mixed and composed together. They are not limited to generator -> runner anymore, any generator can output to any runner/handler/configs/suites it wishes too.
  • Test generation can detect existing tests, and not run generation for those. Effectively the ability to backfill tests.
  • Test generation supports filtering to a specific configuration with a -l <config name> [<config name> [...]] flag.
  • BLS and ssz-generic are moved to general, a folder next to the real configurations. These general tests simply do not require any particular configuration for correctness.
  • Future releases will include the used configuration inside the respective configuration directory, named config.yaml.
  • The idea of fork-timelines is abolished in favor of configurations. Configurations have always been used to specify fork times (e.g. PHASE_1_FORK_EPOCH), and this allows for a less complicated forking setup in the future. As there are no forks yet, this should not affect clients directly. The configuration procedure for forks is more extensively described in the main test format doc.

Note: before introducing an update to the generic-ssz format this PR actually reduced line count. The documentation and generation code however pulled it back. However, more testing and documentation is always welcome :)

@protolambda
Copy link
Contributor Author

Not sure what happened, but it doesn't show up as draft PR anymore. Still a work in progress (most testgen is updated, just need more documentation updates).

Copy link
Contributor

@djrtwo djrtwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome! didn't quite make it through all of the modified generators but about to step AFK.

Just have some minor fixes along with a few rationale questions on little design decisions. great work :)

specs/test_formats/README.md Outdated Show resolved Hide resolved
specs/test_formats/README.md Outdated Show resolved Hide resolved
specs/test_formats/README.md Outdated Show resolved Hide resolved
specs/test_formats/genesis/initialization.md Show resolved Hide resolved
specs/test_formats/genesis/initialization.md Outdated Show resolved Hide resolved
specs/test_formats/sanity/slots.md Show resolved Hide resolved
specs/test_formats/shuffling/README.md Show resolved Hide resolved
specs/test_formats/ssz_generic/README.md Outdated Show resolved Hide resolved
specs/test_formats/ssz_generic/README.md Show resolved Hide resolved
specs/test_formats/ssz_generic/README.md Outdated Show resolved Hide resolved
protolambda and others added 3 commits July 31, 2019 02:02
Co-Authored-By: Danny Ryan <dannyjryan@gmail.com>
Co-Authored-By: Danny Ryan <dannyjryan@gmail.com>
@djrtwo djrtwo merged commit 73a9e86 into v08x Aug 3, 2019
@djrtwo djrtwo deleted the testgen-reorg branch August 3, 2019 12:43
@Kinggeneo83
Copy link

This PR implements the proposed changes to test vector generation in #1311

Change log:

  • Tests are structured as tests/<config name>/<fork or phase name>/<test runner name>/<test handler name>/<test suite name>/<test case>/<output part>. Background details can be found in Test orientation & distribution format issues #1311, and all the necessary information can be found in the new updated specs/test_formats/README.md
  • This effectively reduces resource requirements from running tests to the size of a single test case. This alleviates testing problems presented by Chainsafe and Trinity for the mainnet configuration.
  • For state-transition tests, all data is now also available in SSZ serialized form, as raw bytes in a .ssz file.
  • Test formats have generally not changed, the essential difference is that tests are split into multiple files, down to the pre/post/change level. This enables clients to choose for SSZ encoding over YAML. For better testing speed, and no issues with YAML unmarshalling. This alleviates pains Prysmatic Labs was having with the legacy choice for YAML.
  • A new experimental ssz_generic test generator is included, which utilizes the new ability for differently typed test parts, and starts to fill the previous gap of general-purposes SSZ tests.
  • Test generators can be mixed and composed together. They are not limited to generator -> runner anymore, any generator can output to any runner/handler/configs/suites it wishes too.
  • Test generation can detect existing tests, and not run generation for those. Effectively the ability to backfill tests.
  • Test generation supports filtering to a specific configuration with a -l <config name> [<config name> [...]] flag.
  • BLS and ssz-generic are moved to general, a folder next to the real configurations. These general tests simply do not require any particular configuration for correctness.
  • Future releases will include the used configuration inside the respective configuration directory, named config.yaml.
  • The idea of fork-timelines is abolished in favor of configurations. Configurations have always been used to specify fork times (e.g. PHASE_1_FORK_EPOCH), and this allows for a less complicated forking setup in the future. As there are no forks yet, this should not affect clients directly. The configuration procedure for forks is more extensively described in the main test format doc.

Note: before introducing an update to the generic-ssz format this PR actually reduced line count. The documentation and generation code however pulled it back. However, more testing and documentation is always welcome :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants