Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add caching environment instructions #200

Merged
merged 9 commits into from
Nov 24, 2021
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions .github/workflows/caching-envs-example.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
name: "Caching Env Example"

on:
pull_request:
branches:
- "*"
push:
branches:
- "master"
schedule:
# Note that cronjobs run on master/main by default
- cron: "0 0 * * *"

jobs:
caching-example:
name: Caching

# prevent cronjobs from running on forks
if:
(github.event_name == 'schedule' && github.repository ==
'conda-incubator/setup-miniconda') || (github.event_name != 'schedule')

strategy:
matrix:
include:
- os: ubuntu-latest
label: linux-64
prefix: /usr/share/miniconda3/envs/anaconda-client-env

- os: macos-latest
label: osx-64
prefix: /Users/runner/miniconda3/envs/anaconda-client-env

- os: windows-latest
label: win-64
prefix: C:\Miniconda3\envs\anaconda-client-env

runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v2

- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v2
with:
miniforge-variant: Mambaforge
miniforge-version: latest
activate-environment: anaconda-client-env
use-mamba: true

- name: Set cache date
run: echo "DATE=$(date +'%Y%m%d')" >> $GITHUB_ENV

- name: Cache conda env
uses: actions/cache@v2
with:
path: ${{ matrix.prefix }}
key: ${{ matrix.label }}-conda-${{ hashFiles('etc/example-environment-caching.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
env:
# Increase this value to reset cache if etc/example-environment.yml has not changed
CACHE_NUMBER: 0
id: cache

- name: Update environment
run: mamba env update -n anaconda-client-env -f etc/example-environment-caching.yml
if: steps.cache.outputs.cache-hit != 'true'
69 changes: 69 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -528,6 +528,8 @@ jobs:

## Caching

### Caching packages

If you want to enable package caching for conda you can use the
[cache action](https://github.com/actions/cache) using `~/conda_pkgs_dir` as
path for conda packages.
Expand Down Expand Up @@ -570,6 +572,73 @@ you may want to
[cache those dependencies separately](https://docs.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#caching-dependencies),
as they are not included in the conda package cache.

### Caching environments

A Miniforge variant is recommended to cache deployed environments, since the
Miniconda installation path requires succesive changes of folder ownership in
order to work with the `cache` action.

Every operating system use a different Miniforge `prefix`, so if you want to
cache the environment on all of them you must use a `matrix` strategy.

```yaml
strategy:
matrix:
include:
- os: ubuntu-latest
label: linux-64
prefix: /usr/share/miniconda3/envs/anaconda-client-env

- os: macos-latest
label: osx-64
prefix: /Users/runner/miniconda3/envs/anaconda-client-env

- os: windows-latest
label: win-64
prefix: C:\Miniconda3\envs\anaconda-client-env
```

Then, the first installation step should setup a Miniconda variant without
specifying a environment file.

```yaml
- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v2
with:
miniforge-variant: Mambaforge
miniforge-version: latest
activate-environment: anaconda-client-env
use-mamba: true
```

It's a good idea to refresh the cache every 24 hours to avoid inconsistencies
of package versions between the CI pipeline and local installations. You can
skip this step if you use an environment file product of `conda env export`
or `conda list --explicit`.

```yaml
- name: Set cache date
run: echo "DATE=$(date +'%Y%m%d')" >> $GITHUB_ENV

- uses: actions/cache@v2
with:
path: ${{ matrix.prefix }}
epassaro marked this conversation as resolved.
Show resolved Hide resolved
key: ${{ matrix.label }}-conda-${{ hashFiles('etc/example-environment-caching.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
jaimergp marked this conversation as resolved.
Show resolved Hide resolved
env:
# Increase this value to reset cache if etc/example-environment.yml has not changed
CACHE_NUMBER: 0
id: cache
```

Finally, update the environment based on the environment file if the cache
jaimergp marked this conversation as resolved.
Show resolved Hide resolved
does not exist.

```yaml
- name: Update environment
run: mamba env update -n anaconda-client-env -f etc/example-environment-caching.yml
if: steps.cache.outputs.cache-hit != 'true'
```

### Use a default shell

Assuming you are using the bash shell, now adding to `shell: bash -l {0}` to
Expand Down