Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tune up backend GHA #495

Merged
merged 3 commits into from
Aug 25, 2022
Merged

Tune up backend GHA #495

merged 3 commits into from
Aug 25, 2022

Conversation

sergejparity
Copy link
Contributor

@sergejparity sergejparity commented Aug 25, 2022

Refactored workflow backend.yml by splitting it in four parts. Each job into its independent backend_fmt.yml, backend_check.yml, backend_docs.yml and backend_tests.yml respectively.

The goal of this change is to overcome limitations of the Swatinem/rust-cache GitHub Action. When multiple jobs within one workflow uses above mentioned GHA and executed in parallel it often fails to restore rust cache. Possible two solutions, make jobs using rust-cache to run sequentially within single workflow or setup separate workflows for every job. The latter approach is more reliable as allows to execute jobs in parallel despite little overhead of spinning up additional runners.

As a result workflows execution time reduced almost 3 times. From 15-18 minutes to 5-7 minutes.

@sergejparity sergejparity requested review from a team as code owners August 25, 2022 08:56
@sergejparity sergejparity changed the title [WIP] Tune up backend GHA Tune up backend GHA Aug 25, 2022
Copy link
Collaborator

@jsdw jsdw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit sad that we need to split into multiple files to make thie work better (though well done on reducing the build times so much; that's great :)).

Could you achieve a similar thing by using one file but tagging the rust cache invocation in each job with some different thing to give each a unique cache or something?

Still; I'm ok with it either way; just wondering really because I could see this sort of change being applied across a lot of projects.

@sergejparity
Copy link
Contributor Author

sergejparity commented Aug 25, 2022

Could you achieve a similar thing by using one file but tagging the rust cache invocation in each job with some different thing to give each a unique cache or something?

I've tried that approach as well, but looks like when it happens within one runner those caches corrupting each other. And it always a lottery whether it will work or not. Probably because it populating cache to $workspace/target and in this case we have one workspace - backend. We could checkout repository 3 times into different subfolders, but this would be even more ridiculous setup.

So either in sequence or in separate runners.

@sergejparity sergejparity merged commit fc460d7 into master Aug 25, 2022
@sergejparity sergejparity deleted the sk-tune-backend-gha branch August 25, 2022 15:34
Rose2161 pushed a commit to Cryptob3auty/mtsubstrate-telemen that referenced this pull request Apr 15, 2024
* run cargo steps in sequence

* reorder jobs and gha push event only for master

* split backend gha
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants