Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up CI tests #407

Merged
merged 1 commit into from
Oct 5, 2021
Merged

Speed up CI tests #407

merged 1 commit into from
Oct 5, 2021

Conversation

thesps
Copy link
Contributor

@thesps thesps commented Sep 30, 2021

As we've been adding more tests, the CI testing takes longer to run. The timing varies a lot, but to begin with it would take ~15-20 minutes, while recently with more tests it's taking ~20-30 minutes. This PR speeds things up a bit by submitting each test_*.py file as a separate CI job, which can be assigned to different workers. Previously we just ran all pytests in one job, so each test would run sequentially. As we continue to develop more tests this should help avoid the CI really dragging.

Here the pipeline is generated dynamically: the main .gitlab-ci.yml job runs the generate_ci_yaml.py script, which creates a .yml file defining all the jobs from the available test files. That's uploaded as an artifact and used to trigger the next stage. So to add a new test file you can still just add the test_something.py file to the pytests directory and it will be picked up by the CI.

Here's a view of it running, with some jobs finished and some still going:

Screenshot 2021-09-30 at 17 03 43

The timing seems to now be in the 10-15 minute range, pretty much limited to the time it takes to run the slowest test file - usually the MNIST CNN one.

@vloncar
Copy link
Contributor

vloncar commented Oct 5, 2021

Note: This PR checks for example models in a bit of a hacky way. There is currently no cleaner way to do this, but this will be addressed in the future through an API for accessing example models.

@thesps thesps merged commit ce05499 into master Oct 5, 2021
@thesps thesps deleted the parallel-tests branch October 5, 2021 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants