Reduce execution time of conda workflow #36767
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 tasks
|
Documentation preview for this PR (built with commit 2502f51; changes) is ready! 🎉 |
mkoeppe
changed the title
|
Thanks, that's excellent preparation for #36694, which no longer needs these lines. |
|
Thanks for the review. |
5 tasks
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Dec 10, 2023
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> Reduces the executation time of the conda workflow to 2h, so that it is quicker than the "Build & Test" workflow and devs no longer have to wait 5h until their PR turns green. It also reduces the likelihood that the conda workflows queue up as now all available macos runners are utilized immediately. From sagemath#36694 (comment). <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36767 Reported by: Tobias Diez Reviewer(s):
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Dec 27, 2023
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> The conda workflow is broken since a few months due to various issues that arise from some packages being updated in conda-forge and sage not being quite compatible with the new version. For example, currently it fails as Cython 3.0.3 is used. To prevent such issues happening in the feature, we use [conda- lock](https://conda.github.io/conda-lock/) to create a lock file for each os + python combination, from which one can then create a reproducible conda env. These lock files are created using commands of the form: ```bash conda-lock --channel conda-forge --kind env --platform linux-64 --platform osx-64 --file ./src/environment-dev-3.10.yml --filename- template "src/environment-dev-3.10-{platform}" ``` Follow-up: Add a github workflow that updates these lock files automatically, similar to: - https://github.com/ibis- project/ibis/blob/master/.github/workflows/conda-lock.yml - https://github.com/svopper/munaiah- analyser/blob/main/out/SciTools/iris/.github_workflows_refresh- lockfiles.yml - https://github.com/chipsalliance/fpga-interchange- tests/blob/main/.github/workflows/update_conda_lock.yml - https://github.com/ESMValGroup/ESMValTool/blob/main/.github/workflows/ create-condalock-file.yml ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies - sagemath#36405: to get env files for each python version - sagemath#36767 <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#35986 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik, Isuru Fernando, Matthias Köppe, Tobias Diez
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Jan 2, 2024
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> The conda workflow is broken since a few months due to various issues that arise from some packages being updated in conda-forge and sage not being quite compatible with the new version. For example, currently it fails as Cython 3.0.3 is used. To prevent such issues happening in the feature, we use [conda- lock](https://conda.github.io/conda-lock/) to create a lock file for each os + python combination, from which one can then create a reproducible conda env. These lock files are created using commands of the form: ```bash conda-lock --channel conda-forge --kind env --platform linux-64 --platform osx-64 --file ./src/environment-dev-3.10.yml --filename- template "src/environment-dev-3.10-{platform}" ``` Follow-up: Add a github workflow that updates these lock files automatically, similar to: - https://github.com/ibis- project/ibis/blob/master/.github/workflows/conda-lock.yml - https://github.com/svopper/munaiah- analyser/blob/main/out/SciTools/iris/.github_workflows_refresh- lockfiles.yml - https://github.com/chipsalliance/fpga-interchange- tests/blob/main/.github/workflows/update_conda_lock.yml - https://github.com/ESMValGroup/ESMValTool/blob/main/.github/workflows/ create-condalock-file.yml ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies - sagemath#36405: to get env files for each python version - sagemath#36767 <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#35986 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik, Isuru Fernando, Matthias Köppe, Tobias Diez
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Jan 5, 2024
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> The conda workflow is broken since a few months due to various issues that arise from some packages being updated in conda-forge and sage not being quite compatible with the new version. For example, currently it fails as Cython 3.0.3 is used. To prevent such issues happening in the feature, we use [conda- lock](https://conda.github.io/conda-lock/) to create a lock file for each os + python combination, from which one can then create a reproducible conda env. These lock files are created using commands of the form: ```bash conda-lock --channel conda-forge --kind env --platform linux-64 --platform osx-64 --file ./src/environment-dev-3.10.yml --filename- template "src/environment-dev-3.10-{platform}" ``` Follow-up: Add a github workflow that updates these lock files automatically, similar to: - https://github.com/ibis- project/ibis/blob/master/.github/workflows/conda-lock.yml - https://github.com/svopper/munaiah- analyser/blob/main/out/SciTools/iris/.github_workflows_refresh- lockfiles.yml - https://github.com/chipsalliance/fpga-interchange- tests/blob/main/.github/workflows/update_conda_lock.yml - https://github.com/ESMValGroup/ESMValTool/blob/main/.github/workflows/ create-condalock-file.yml ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies - sagemath#36405: to get env files for each python version - sagemath#36767 <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#35986 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik, Isuru Fernando, Matthias Köppe, Tobias Diez
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Jan 13, 2024
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> The conda workflow is broken since a few months due to various issues that arise from some packages being updated in conda-forge and sage not being quite compatible with the new version. For example, currently it fails as Cython 3.0.3 is used. To prevent such issues happening in the feature, we use [conda- lock](https://conda.github.io/conda-lock/) to create a lock file for each os + python combination, from which one can then create a reproducible conda env. These lock files are created using commands of the form: ```bash conda-lock --channel conda-forge --kind env --platform linux-64 --platform osx-64 --file ./src/environment-dev-3.10.yml --filename- template "src/environment-dev-3.10-{platform}" ``` Follow-up: Add a github workflow that updates these lock files automatically, similar to: - https://github.com/ibis- project/ibis/blob/master/.github/workflows/conda-lock.yml - https://github.com/svopper/munaiah- analyser/blob/main/out/SciTools/iris/.github_workflows_refresh- lockfiles.yml - https://github.com/chipsalliance/fpga-interchange- tests/blob/main/.github/workflows/update_conda_lock.yml - https://github.com/ESMValGroup/ESMValTool/blob/main/.github/workflows/ create-condalock-file.yml ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies - sagemath#36405: to get env files for each python version - sagemath#36767 <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#35986 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik, Isuru Fernando, Matthias Köppe, Tobias Diez
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reduces the executation time of the conda workflow to 2h, so that it is quicker than the "Build & Test" workflow and devs no longer have to wait 5h until their PR turns green. It also reduces the likelihood that the conda workflows queue up as now all available macos runners are utilized immediately.
From #36694 (comment).
📝 Checklist
⌛ Dependencies