-
Notifications
You must be signed in to change notification settings - Fork 41
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
End-to-End System Level Testing Maturity Model (#1332)
maturity model for e2e
- Loading branch information
Showing
10 changed files
with
148 additions
and
103 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,21 +1,5 @@ | ||
| # Testing Maturity Model | ||
|
|
||
| [Here](https://github.com/smartcontractkit/chainlink-testing-framework/actions/runs/11739154666/job/32703095118?pr=1311) are our "golden" templates for end-to-end tests, covering every test type: | ||
| Read this [doc](https://github.com/smartcontractkit/chainlink-testing-framework/tree/main/framework/examples/myproject) to understand the rationale behind our testing approach and to explore the stages of maturity in end-to-end testing. | ||
|
|
||
| - `Smoke` | ||
| - `PerformanceBaseline` | ||
| - `Chaos` | ||
| - `Upgrade` | ||
|
|
||
| These tests act as a maturity model and are implemented across all our products. | ||
|
|
||
| Refer to this README to understand the rationale behind our testing approach and to explore the stages of maturity in end-to-end testing. | ||
|
|
||
| ## Developing | ||
| Run the tests locally | ||
| ``` | ||
| CTF_CONFIGS=smoke.toml go test -v -run TestSmoke | ||
| CTF_CONFIGS=performance_baseline.toml go test -v -run TestPerformanceBaseline | ||
| CTF_CONFIGS=chaos.toml go test -v -run TestChaos | ||
| CTF_CONFIGS=upgrade.toml go test -v -run TestUpgrade | ||
| ``` | ||
| The following chapters detail specific testing types and best practices for system-level end-to-end testing. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| - Switch to port checking when we boot Chainlink nodes (eliminates chaos flakes) | ||
| - Add initial e2e maturity model pipeline |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,10 +1,54 @@ | ||
| # End-to-End Testing Project Maturity Model | ||
|
|
||
| [](https://github.com/smartcontractkit/chainlink-testing-framework/actions/workflows/framework-golden-tests.yml) | ||
| [](https://github.com/smartcontractkit/chainlink-testing-framework/actions/workflows/framework-golden-tests.yml) | ||
| [](https://github.com/smartcontractkit/chainlink-testing-framework/actions/workflows/framework-golden-tests.yml) | ||
| [](https://github.com/smartcontractkit/chainlink-testing-framework/actions/workflows/framework-golden-tests.yml) | ||
| [](https://github.com/smartcontractkit/chainlink-testing-framework/actions/workflows/framework-golden-tests.yml) | ||
|
|
||
| ## Level 1 | ||
| The team maintains a system-level smoke test where all components are deployed using `docker`. | ||
|
|
||
| All on-chain changes are done through [chainlink-deployments](https://github.com/smartcontractkit/chainlink-deployments). | ||
|
|
||
| The test is readable, and the README clearly explains its purpose. | ||
|
|
||
| The test is reliable and stable when run with a `-count 30`. | ||
|
|
||
| If your project includes multiple use cases and functionality suitable for end-to-end testing, you can add additional tests at this level. | ||
|
|
||
| ## Level 2 | ||
| The team has an "upgrade" test to verify product compatibility with older versions. | ||
|
|
||
| While the number of compatible versions is team-determined, identifying incompatibilities at the system level early is a valuable, mature practice. | ||
|
|
||
| This test deploys specific platform and plugin versions, performs an end-to-end smoke test, and then upgrades (or migrates) the plugin(s) or platform on the same database to ensure that users remain unaffected by the upgrade. | ||
|
|
||
| ## Level 3 | ||
| The team has a baseline performance testing suite. | ||
|
|
||
| At this level, the focus is not on improving performance but on detecting any performance degradation, supported by a reliable CI pipeline. | ||
|
|
||
| This pipeline runs as needed—nightly or before releases—enabling early detection of performance issues across all critical on-chain and off-chain functionality. | ||
|
|
||
| This stage combines performance testing with observability enhancements. The team should have fundamental system-level performance tests in place, along with dashboards to monitor product-specific metrics. | ||
|
|
||
| ## Level 4 | ||
| The team incorporates chaos engineering practices to test the system’s failure modes. | ||
|
|
||
| This stage builds on [Level 3](#level-3), as it not only verifies that the system is reliable and can recover from reasonable failures but also ensures an understanding of how these failures impact performance and user experience. | ||
|
|
||
| ## Level 5 | ||
| The team has complete ownership of their persistent staging environment. | ||
|
|
||
| They can perform upgrades, data migrations, and run advanced load tests to validate the scalability of their applications. | ||
|
|
||
| ## Developing | ||
| Run the tests locally | ||
| ``` | ||
| CTF_CONFIGS=smoke.toml go test -v -run TestSmoke | ||
| CTF_CONFIGS=upgrade.toml go test -v -run TestUpgrade | ||
| CTF_CONFIGS=performance_baseline.toml go test -v -run TestPerformanceBaseline | ||
| CTF_CONFIGS=chaos.toml go test -v -run TestChaos | ||
| CTF_CONFIGS=upgrade.toml go test -v -run TestUpgrade | ||
| CTF_CONFIGS=scalability.toml go test -v -run TestScalability | ||
| ``` |
2 changes: 1 addition & 1 deletion
2
framework/examples/myproject/gun.go → ...work/examples/myproject/generators/gun.go
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.