numpy unable to import packages with python distroless

[justmike1]

Describe the bug
My python app uses numpy version 1.23.4, it fails on importing C-extensions.
Numpy is also very popular package, the python packages should come with prerequisites in order for it to work properly.

To Reproduce
Python app with numpy usage.

Expected behavior
Numpy imports it's packages and works

Console Output
If applicable, add information from your container run

Traceback (most recent call last):
  File "/app/server/app.py", line 8, in <module>
    import ai_recsys as recsys
  File "/app/server/ai_recsys.py", line 11, in <module>
    import pandas as pd
  File "/usr/local/lib/python3.8/site-packages/pandas/__init__.py", line 16, in <module>
    raise ImportError(
ImportError: Unable to import required dependencies:
numpy: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.9 from "/usr/bin/python3.9"
  * The NumPy version is: "1.23.4"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

Additional context
Dockerfile:

# Build stage
FROM python:3.8-slim-buster as build

ENV PB_REL="https://github.com/protocolbuffers/protobuf/releases"
ENV VERSION="21.12"
ENV PYTHONUNBUFFERED=true
WORKDIR /protobuf
COPY server/proto .

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    unzip ca-certificates build-essential default-libmysqlclient-dev curl zip && \
    rm -rf /var/lib/apt/lists/*

RUN curl -LO $PB_REL/download/v${VERSION}/protoc-${VERSION}-linux-x86_64.zip && \
    unzip -o protoc-${VERSION}-linux-x86_64.zip -d /usr/local bin/protoc && \
    unzip -o protoc-${VERSION}-linux-x86_64.zip -d /usr/local include/* && \
    rm -rf protoc-${VERSION}-linux-x86_64.zip

RUN mkdir output/ && \
    for p in $(find . -type f -name "*.proto"); do \
      protoc --python_out=output/ --pyi_out=output/ ${p} ; \
    done

WORKDIR /app
COPY server server
RUN mv /protobuf/output/** server/.
COPY Pipfile* ./

RUN pip install pipenv && \
  pipenv install --verbose --system --deploy --ignore-pipfile

# Final stage
FROM gcr.io/distroless/python3
COPY --from=build /usr/local/lib/python3.8/site-packages /usr/local/lib/python3.8/site-packages
COPY --from=build /app /app
WORKDIR /app
ENV PYTHONPATH=/usr/local/lib/python3.8/site-packages

CMD ["server/app.py"]

[dlorenc]

I gave it a try on the chainguard image and it seems to work:

$ docker run -it -u root --entrypoint=sh cgr.dev/chainguard/python:latest-dev
$ pip install numpy pytest hypothesis typing_extensions
$ python
# import numpy
# numpy.test()
25189 passed, 411 skipped, 1307 deselected, 31 xfailed, 5 xpassed, 25 warnings in 66.86s (0:01:06)

[justmike1]

but chainguard uses python3.11 which is incompatible with my app, and probably many others. @dlorenc

[dlorenc]

but chainguard uses python3.11 which is incompatible with my app, and probably many others. @dlorenc

Which version are you looking for? We have 3.10 as well, and others are on the way.

[justmike1]

@dlorenc
I am working on 3.8 (as you see in the Dockerfile), I didn't test newer versions

[justmike1]

it worked using this Dockerfile:

FROM python:3.8-slim-buster AS protobuf

ENV PB_REL="https://github.com/protocolbuffers/protobuf/releases"
ENV VERSION="21.12"
ENV PYTHONUNBUFFERED=true
WORKDIR /protobuf
COPY server/proto .

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    unzip curl zip && \
    rm -rf /var/lib/apt/lists/*

RUN curl -LO $PB_REL/download/v${VERSION}/protoc-${VERSION}-linux-x86_64.zip && \
    unzip -o protoc-${VERSION}-linux-x86_64.zip -d /usr/local bin/protoc && \
    unzip -o protoc-${VERSION}-linux-x86_64.zip -d /usr/local include/* && \
    rm -rf protoc-${VERSION}-linux-x86_64.zip

RUN mkdir output/ && \
    for p in $(find . -type f -name "*.proto"); do \
      protoc --python_out=output/ --pyi_out=output/ ${p} ; \
    done

COPY Pipfile* ./
RUN pip install pipenv && \
    pipenv requirements > requirements.txt

FROM cgr.dev/chainguard/python:3.10-dev AS build
WORKDIR /app
COPY --from=protobuf /protobuf/requirements.txt .

RUN pip install -r requirements.txt --user

FROM cgr.dev/chainguard/python:3.10
COPY --from=build /home/nonroot/.local/lib/python3.10/site-packages /home/nonroot/.local/lib/python3.10/site-packages
COPY server /app/server
COPY --from=protobuf /protobuf/output /app/server/.
WORKDIR /app
USER root
ENV PYTHONPATH=/home/nonroot/.local/lib/python3.10/site-packages
CMD ["server/app.py"]

[justmike1]

I would like pipenv to be added to chainguard, but obviously that's just my egoistic liking, this works fine :) @dlorenc

[justmike1]

BTW this has reduced my vulnerabilities from 192 to 0, I can say Velotix organization is pleased, thank you for the good work

[dlorenc]

Yay! Can you file an issue for us here? https://github.com/wolfi-dev/os

[justmike1]

done @dlorenc