Docker multi-stage build Dockerfile best practices #1178
Comments
hozn
changed the title
hozn
changed the title
|
I too have this exact use case (the short-sighted one: use an intermediate image to generate wheels); I could not find how to do this with poetry. I haven't been able to create wheels of all dependencies, but creating a wheel of my main package does work with |
|
Yeah, this and a couple other issues caused us to switch back to good-old pip/setuptools, which at least provides the flexibility to support our development workflow -- if a bit less elegant. |
|
We've had relatively good success copying virtualenvs between images with pip. I'm just beginning to see if we can transition to poetry - we've just been using it for small utilities right now but we're using the following code to do what you're describing: FROM python:3.7.4-slim as python-base
ENV PIP_NO_CACHE_DIR=off \
PIP_DISABLE_PIP_VERSION_CHECK=on \
PIP_DEFAULT_TIMEOUT=100 \
POETRY_PATH=/opt/poetry \
VENV_PATH=/opt/venv \
POETRY_VERSION=0.12.17
ENV PATH="$POETRY_PATH/bin:$VENV_PATH/bin:$PATH"
FROM python-base as poetry
RUN apt-get update \
&& apt-get install --no-install-recommends -y \
# deps for installing poetry
curl \
# deps for building python deps
build-essential \
\
# install poetry - uses $POETRY_VERSION internally
&& curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python \
&& mv /root/.poetry $POETRY_PATH \
&& poetry --version \
\
# configure poetry & make a virtualenv ahead of time since we only need one
&& python -m venv $VENV_PATH \
&& poetry config settings.virtualenvs.create false \
\
# cleanup
&& rm -rf /var/lib/apt/lists/*
COPY poetry.lock pyproject.toml ./
RUN poetry install --no-interaction --no-ansi -vvv
FROM python-base as runtime
WORKDIR /app
COPY --from=poetry $VENV_PATH $VENV_PATH
COPY . ./
ENTRYPOINT ["python", "-m", "app"]Haven't figured out a clean way to work with prod vs. dev dependencies |
|
Thanks, @dpraul , this is a very helpful reference. |
|
No problem! Disclaimer: not heavily tested, so YMMV. Open on suggestions for how to improve it |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
Closing this issue automatically because it has not had any activity since it has been marked as stale. If you think it is still relevant and should be addressed, feel free to open a new one. |
|
https://github.com/Docker-s-IMAGES/no-pypoetry FROM abersh/no-pypoetry as requirements
FROM python:3.7
# ... yourself commands
COPY --from=requirements /src/requirements.txt .
RUN pip install -r requirements.txt
# ... yourself commandsYou can use this multi-stage Dockerfile to ensure that you don't need to install a poetry in your final image. This can reduce the installation of many useless dependencies. |
|
Hey, I found this solution for me to work best: FROM python:3.9-alpine AS builder
WORKDIR /app
ADD pyproject.toml poetry.lock /app/
RUN apk add build-base libffi-dev
RUN pip install poetry
RUN poetry config virtualenvs.in-project true
RUN poetry install --no-ansi
# ---
FROM python:3.9-alpine
WORKDIR /app
COPY --from=builder /app /app
ADD . /app
RUN addgroup -g 1000 app
RUN adduser app -h /app -u 1000 -G 1000 -DH
USER 1000
# change this to match your application
CMD /app/.venv/bin/python -m module_name
# or
CMD /app/.venv/bin/python app.pyDont forget a Ticks all my boxes:
Just make sure to use the same path in the builder and the final image, virtualenv uses some hardcoded paths. Change the EDIT: Updated to reflect some of @alexpovel's critics |
|
Building on top of @foosinn 's approach: # Global ARG, available to all stages (if renewed)
ARG WORKDIR="/app"
FROM python:3.10 AS builder
# Renew (https://stackoverflow.com/a/53682110):
ARG WORKDIR
# Don't buffer `stdout`:
ENV PYTHONUNBUFFERED=1
# Don't create `.pyc` files:
ENV PYTHONDONTWRITEBYTECODE=1
RUN pip install poetry && poetry config virtualenvs.in-project true
WORKDIR ${WORKDIR}
COPY . .
RUN poetry install --only main
FROM python:3.10-alpine
ARG WORKDIR
WORKDIR ${WORKDIR}
COPY --from=builder ${WORKDIR} .
# For options, see https://boxmatrix.info/wiki/Property:adduser
RUN adduser app -DHh ${WORKDIR} -u 1000
USER 1000
# App-specific settings:
EXPOSE 8080
ENTRYPOINT [ "./.venv/bin/python", "-m", "ancv" ]
CMD [ "serve", "api", "--port", "8080" ]from here. As opposed to simply using Changes from the previous post:
|
|
Thanks for bumping this -- the pattern you describe here is another one to polish and roll up into the work described at #6398 (which is mostly about containers despite ostensibly being not specific to them, as containers are usually the only place install into system site-packages comes up). |
|
@alexpovel thanks for you input. I've updated some points where i fully agree. thanks a lot! |
|
I provide two docker images that you can use as builder base for multistage builds. They also contain an example application which demonstrates multistage builds with Poetry:
I aim for the best practices there. Please check them out and leave a comment if you have additional suggestions. |
|
@foosinn thanks that helped me a lot. I had to modify the adduser / addgroup lines, and personally modified mine for flask. FROM python:3.10-alpine AS builder
WORKDIR /app
ADD pyproject.toml poetry.lock /app/
RUN apk add build-base libffi-dev
RUN pip install poetry
RUN poetry config virtualenvs.in-project true
RUN poetry install --no-ansi
FROM python:3.10-alpine
WORKDIR /app
COPY --from=builder /app /app
ADD . /app
# Create a new group `app` with Group ID `1000`.
RUN addgroup --gid 1000 app
# Create a new user `app`, sets home directory to `/app`, User ID `1000`, in
# the group `app`. The `-DH` option results in a system account.
RUN adduser app -h /app -u 1000 -G app -DH
# Change the user for subsequent commands in Dockerfile to the user with ID
# `1000`.
USER 1000
CMD ["/app/.venv/bin/gunicorn", "--bind", ":80", "app:app"] |
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Question
I continue to try to replace our pip/setuptools-based system with poetry, but hit a new snag when it comes to how we build our Docker images.
Here's the basic pattern we use for our Docker images, in a build and a deploy stage:
Here's how this translates into a Dockerfile:
How do I do this with poetry? -- in the most short-sighted form, I'd like to know how to collect all dependencies as wheels in order to match this pattern.
However, my real requirement here is just to have separate build and deploy stages where the deploy image has no python (or lower-level) build-related tools installed (but does have pip) and simply takes artifacts from the build image.
(I suppose one idea would be to treat the entire virtualenv from the build stage as an artifact? That seems a little dirty, but provided the base OS images were the same, might work?)
The text was updated successfully, but these errors were encountered: