CI: yaml-test-generation trigger

[protolambda]

To make good use of the test-generation from #851, we need to output the generated tests to an repository. Similar to the eth2.0-test-generators -> eth2.0-tests workflow, but with the following improvements:

• Use Git LFS for the yaml test vectors repository. With the large increase in tests, and test-suite size, we need something that can host ~1 gigabyte of files (See ETH 1.0 tests as size reference). Use of Git LFS improves cloning and git speed. We're not interested in diffing on github (you can still do so offline), as it's all generated code anyway, we're not making changes to it directly.
• Versioning based on combination of spec-version (vA.B.C), and an extra number X based on any changes in the generation of tests. Proposed format: vA.B.C-genX example: v0.6.1-gen2. Clients can safely upgrade from one gen to another if A.B.C is still the same. I.e. tests are improved/extended, without functionality changes.

What to implement:

• Create a new repository for test outputs, with Git LFS from the start. Proposed name: eth2.0-spec-tests
• Deprecate the old eth2.0-tests and eth2.0-test-generators repositories
  • To summarize the transition away from eth2.0-test-generators:
    • it does not have consistent clean access to the most up-to-date pyspec package
    • introduces an overhead on versioning
    • cannot be updated in sync with specs when viable
    • test-writer life could be much easier with all tooling available in Combine specs and test-generators #851 (configs for forks, constant presets, base-generator, utility packages, pyspec)
• eth2.0-tests is just deprecated. The new format is not that different when you look at existing features, but improves structuring, and scales better to larger amounts of tests
• Update the CI script to:
  • on every merge from eth2.0-specs/dev into eth2.0-specs/test-gen:
    • build the tests make gen_yaml_tests
    • push the output to eth2.0-spec-tests/master (git commit, LFS is automatic if activated correctly)
  • We manually create git tags (or use github releases, which does the same) to version the eth2.0-spec-tests repository. It's just a matter of git tag -S -a v1.2.3-gen4 && git push v1.2.3-gen4

I believe this test-gen branch based trigger works best, as it does not create a dependency on the release cycle, and makes it easy to re-trigger in case of a problem. The master branch on the eth2.0-spec-tests repository will just not be as frozen as the one in the specs. We'll have spec-tests releases anyway, so that doesn't matter.

[djrtwo]

How do you want to release tests upon formal releases? Just manually?

How do you want to handle the case where a bug is fixed in master and back-ported to dev? In this case we would want to release tests against the master which might have no dev counterpart

[protolambda]

How do you want to release tests upon formal releases?

Create a release (annotated signed git tag) on the tests repo, on the master branch. Yes, manually. It's a very small task, that only repeats every ~2-3 weeks.

How do you want to handle the case where a bug is fixed in master and back-ported to dev?
Good point, we may need to create a separate trigger to update the tests-repo master branch, without going from dev.

We can make it work like this:

• specs#dev -> specs#dev-tests triggers a CI job that outputs to tests#dev (branched of tests#dev)
• specs#master > specs#fix-tests triggers a CI job that outputs to tests#fix (branched of tests#master)

Workflow:

• on specs PR, just merge from feature branch into dev, no CI trigger
• when dev is considered to have working test-generation, do a specs#dev -> specs#dev-tests, and try the tests with a client/spec that is up to date with dev (ZRNT could do this)
• for a specs release, we do a specs#dev -> specs#dev-tests, and then merge the CI output from tests#dev to tests#master, and mark it as release
• for a specs fix on master, we do a specs#master > specs#fix-tests, which builds the tests for the current master branch, enables us to check the resulting tests#fix branch, and then we merge tests#fix to tests#master

[protolambda]

See PR #968

[protolambda]

Outdated. We've different configurations/modes now. CI runs tests, and state-transition generators are unified, so no need to run for ensuring that they work, as the they are already running in test-mode. The few other generators don't change as much. It's trade-offs around, but it works, and gets us fast CI, and working large size test-vector generation. Closing this issue.