CI: move time consuming tests to separate workflow that runs nightly #5354
Conversation
As mentioned, the only issue here is that, this is can going to kill code coverage of unit tests Even if we do add some, I don't think we should mark all migration tests as nightly. It should at most be: and the tests under i.e. only ones that relate to "legacy" migrations |
I don't think this should be a problem though. Sure the coverage percentage reported by Codecov will be a few points lower, but we know that the code is still being covered, just not at every commit. That is the whole point of this change though. We discussed and agreed that the tests were starting to take way too long and it would make sense to split off those tests that don't require being run at every commit. I don't think we should be led in this discussion by the percentage reported by Codecov. That is just an indicator.
That is fair enough. But that is essentially the entire directory which I mark now, except for |
We are never going to get 100% test coverage with that attitude 😛
Yeh sounds good |
Just have to wait until we drop the legacy migrations 😄 |
The test that runs the Reverse Polish Notation (RPN) workchains and the migrations tests are relatively time consuming and they should not be affected by most commits. That's why we can afford to have them just run nightly instead of on every PR. A `nightly` GHA workflow is added that runs every day at midnight. It runs a bash script `tests_nightly.sh` that in turns runs the script `test_polish_workchains.sh` as well as the unit test using `pytest`. The latter is called with the option `-m 'nightly'` which will select only tests that are marked as nightly. Most tests under `tests/backends/aiida_sqlalchemy/migrations` are marked as nightly. This is done by applying the `@pytest.mark.nightly` decorator directly to the test functions. For the tests under the `django_branch` and `sqlalchemy_branch` this is instead done dynamically through the `pytest_collection_modifyitems` function in the `conftest.py` module since having to manually apply this to all test would be too tedious.
sphuber
force-pushed
the
fix/ci-add-nightly
branch
from
34c9088 to
78e981e
Compare
Done! |
The test that runs the Reverse Polish Notation (RPN) workchains and the
migrations tests are relatively time consuming and they should not be
affected by most commits. That's why we can afford to have them just run
nightly instead of on every PR.
A
nightlyGHA workflow is added that runs every day at midnight. Itruns a bash script
tests_nightly.shthat in turns runs the scripttest_polish_workchains.shas well as the unit test usingpytest. Thelatter is called with the option
-m 'nightly'which will select onlytests that are marked as nightly.
All tests under
tests/backends/aiida_sqlalchemy/migrationsare markedas nightly dynamically through the
pytest_collection_modifyitemsfunction in the
tests/backends/aiida_sqlalchemy/migrations/conftest.pyconfiguration file.