Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support conda/mamba install --copy #267

Merged
merged 2 commits into from
Nov 15, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 24 additions & 4 deletions conda_lock/conda_lock.py
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,11 @@ def do_validate_platform(lockfile: str) -> None:


def do_conda_install(
conda: PathLike, prefix: str, name: str, file: pathlib.Path
conda: PathLike,
prefix: "str | None",
name: "str | None",
file: pathlib.Path,
copy: bool,
) -> None:

_conda = partial(_invoke_conda, conda, prefix, name, check_call=True)
Expand All @@ -203,13 +207,18 @@ def do_conda_install(
else:
pip_requirements = []

env_prefix = ["env"] if kind == "env" and not is_micromamba(conda) else []
copy_arg = ["--copy"] if kind != "env" and copy else []
yes_arg = ["--yes"] if kind != "env" else []

_conda(
[
*(["env"] if kind == "env" and not is_micromamba(conda) else []),
*env_prefix,
"create",
*copy_arg,
"--file",
str(file),
*([] if kind == "env" else ["--yes"]),
*yes_arg,
],
)

Expand Down Expand Up @@ -1206,6 +1215,14 @@ def lock(
default=False,
help="don't attempt to use or install micromamba.",
)
@click.option(
"--copy",
is_flag=True,
help=(
"Install using `--copy` to prevent links. "
"This is useful for building containers"
),
)
@click.option("-p", "--prefix", help="Full path to environment location (i.e. prefix).")
@click.option("-n", "--name", help="Name of environment.")
@click.option(
Expand Down Expand Up @@ -1247,6 +1264,7 @@ def install(
conda: Optional[str],
mamba: bool,
micromamba: bool,
copy: bool,
prefix: Optional[str],
name: Optional[str],
lock_file: pathlib.Path,
Expand All @@ -1269,7 +1287,9 @@ def install(
yaml.safe_load(auth) if auth else read_json(auth_file) if auth_file else None
)
_conda_exe = determine_conda_executable(conda, mamba=mamba, micromamba=micromamba)
install_func = partial(do_conda_install, conda=_conda_exe, prefix=prefix, name=name)
install_func = partial(
do_conda_install, conda=_conda_exe, prefix=prefix, name=name, copy=copy
)
if validate_platform and not lock_file.name.endswith(DEFAULT_LOCKFILE_NAME):
lockfile_contents = read_file(lock_file)
try:
Expand Down
8 changes: 5 additions & 3 deletions conda_lock/invoke_conda.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,8 @@ def determine_conda_executable(

def _invoke_conda(
conda: PathLike,
prefix: str,
name: str,
prefix: "str | None",
name: "str | None",
command_args: Sequence[PathLike],
post_args: Sequence[PathLike] = [],
check_call: bool = False,
Expand Down Expand Up @@ -95,9 +95,11 @@ def _invoke_conda(
if prefix:
common_args.append("--prefix")
common_args.append(prefix)
if name:
elif name:
common_args.append("--name")
common_args.append(name)
else:
raise ValueError("Neither prefix, nor name provided.")
conda_flags = os.environ.get("CONDA_FLAGS")
if conda_flags:
common_args.extend(shlex.split(conda_flags))
Expand Down
65 changes: 58 additions & 7 deletions docs/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ In order to use conda-lock in a docker-style context you want to add the lockfil
docker container. This avoids the case where the `conda install xyz` statement is cached.

Given a file tree like

```
Dockerfile
environment.yaml
Expand All @@ -15,24 +16,74 @@ You want a dockerfile that is structured something similar to this
```Dockerfile
# Dockerfile

# Build container
FROM continuumio/miniconda:latest as conda
# -----------------
# Builder container
# -----------------
FROM continuumio/miniconda:latest as builder

ADD conda-linux-64.lock /locks/conda-linux-64.lock
COPY conda-linux-64.lock /locks/conda-linux-64.lock
RUN conda create -p /opt/env --copy --file /locks/conda-linux-64.lock

# -----------------
# Primary container
# -----------------
FROM gcr.io/distroless/base-debian10
# copy over the generated environment
COPY --from=conda /opt/env /opt/env
COPY --from=builder /opt/env /opt/env
ENV PATH="/opt/env/bin:${PATH}"
```

To get this to work nicely generate the platform specific lock run
To get this to work nicely generate the platform specific lock run something
like this in your shell

```shell
```bash
# Update the lockfile
conda lock --format explicit --platform linux-64
docker build .
# build the image
docker build -t myimagename:mytag .
```

This will ensure that your conda dependencies used in this docker container are
always exactly reproducible.

## conda-lock inside a build container

You can also use conda-lock with a build-container style system if you make use of
the `--copy` flag from `conda-lock install`

```Dockerfile
# -----------------
# Builder container
# -----------------
FROM condaforge/mambaforge:4.14.0-0 as builder

COPY environment.yml /docker/environment.yml

RUN . /opt/conda/etc/profile.d/conda.sh && \
mamba create --name lock && \
conda activate lock && \
mamba env list && \
mamba install --yes pip conda-lock>=1.2.2 setuptools wheel && \
conda-lock lock \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I'm following correctly, then conda-lock.yml only lives in the Docker cache where it can't be tracked with version control. Do we really want to recommend this?

Copy link
Collaborator Author

@mariusvniekerk mariusvniekerk Nov 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hence the two approaches in this document

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like I'm missing something obvious here... I think I don't know what is a "build-container style system".

Two reasons I can think of off the top of my head for locking:

  1. Control explicitly when dependencies are updated
  2. Having an explicit record of the versions installed

It seems like neither of these applies in this case. (For 2. at least there's a file in the container, but if you have to go into the container to see it, you might as well run conda env export.)

What's the benefit to this approach over installing directly from environment.yml?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the main benefit is more when you are using something like earthly and have a single earthfile specify multiple docker outputs.

  1. This example does NOT deal with this case
  2. The explicit record is at least kept in the build-container style build described here, which means you can introspect a bit more easily as to what should be in the env.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I didn't know about Earthly. So if I understand correctly, one benefit is that the lockfile can be reused in multiple Dockerfile stages (or similar) to avoid multiple solves.

--platform linux-64 \
--file /docker/environment.yml \
--kind lock \
--lockfile /docker/conda-lock.yml

RUN . /opt/conda/etc/profile.d/conda.sh && \
conda activate lock && \
conda-lock install \
--mamba \
--copy \
--prefix /opt/env \
/docker/conda-lock.yml
# optionally you can perfom some more cleanup on your conda install after this
# to get a leaner conda environment
Comment on lines +80 to +81
Copy link
Contributor

@maresb maresb Oct 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's just be explicit here?

Suggested change
# optionally you can perfom some more cleanup on your conda install after this
# to get a leaner conda environment
RUN mamba clean --all --force-pkgs-dirs --yes


# -----------------
# Primary container
# -----------------
FROM gcr.io/distroless/base-debian10
COPY --from=builder /opt/env /opt/env
ENV PATH="/opt/env/bin:${PATH}"
```