Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deps: bump vllm-tgis-adapter to 0.2.4 #126

Closed
wants to merge 5 commits into from
Closed

deps: bump vllm-tgis-adapter to 0.2.4 #126

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a58d5f2
deps: bump vllm-tgis-adapter to 0.2.4
dtrifiro Aug 12, 2024
5f322d4
Dockerfile: use uv pip everywhere (it's faster)
dtrifiro Aug 12, 2024
6100606
Dockerfile.ubi: cuda-base: add missing runtime deps
dtrifiro Aug 12, 2024
ad5028e
Dockerfile.ubi: change release stage base to python-cuda-base
dtrifiro Aug 12, 2024
c09e8f9
Dockerfile.ubi: add yum cache bind mount where possible
dtrifiro Aug 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 37 additions & 17 deletions Dockerfile.ubi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@ ARG TORCH_CUDA_ARCH_LIST="7.0 7.5 8.0 8.6 8.9 9.0+PTX"
FROM registry.access.redhat.com/ubi9/ubi-minimal:${BASE_UBI_IMAGE_TAG} as base
ARG PYTHON_VERSION

RUN microdnf install -y \
python${PYTHON_VERSION}-pip python${PYTHON_VERSION}-wheel \
RUN --mount=type=cache,target=/var/cache/yum \
microdnf install -y \
python${PYTHON_VERSION}-pip python${PYTHON_VERSION}-wheel \
&& microdnf clean all

WORKDIR /workspace
Expand All @@ -18,7 +19,8 @@ ENV LANG=C.UTF-8 \
LC_ALL=C.UTF-8

# Some utils for dev purposes - tar required for kubectl cp
RUN microdnf install -y \
RUN --mount=type=cache,target=/var/cache/yum \
microdnf install -y \
which procps findutils tar vim git\
&& microdnf clean all

Expand All @@ -30,9 +32,12 @@ ARG PYTHON_VERSION

ENV VIRTUAL_ENV=/opt/vllm
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
RUN microdnf install -y \
python${PYTHON_VERSION}-devel python${PYTHON_VERSION}-pip python${PYTHON_VERSION}-wheel && \
python${PYTHON_VERSION} -m venv $VIRTUAL_ENV && pip install --no-cache -U pip wheel && microdnf clean all

RUN --mount=type=cache,target=/var/cache/yum \
--mount=type=cache,target=/root/.cache/pip \
microdnf install -y \
python${PYTHON_VERSION}-devel python${PYTHON_VERSION}-pip python${PYTHON_VERSION}-wheel && \
python${PYTHON_VERSION} -m venv $VIRTUAL_ENV && pip install --no-cache -U pip wheel uv && microdnf clean all


## CUDA Base ###################################################################
Expand All @@ -41,9 +46,14 @@ FROM python-install as cuda-base
RUN curl -Lo /etc/yum.repos.d/cuda-rhel9.repo \
https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo

RUN microdnf install -y \
cuda-nvcc-12-4 cuda-nvtx-12-4 cuda-libraries-devel-12-4 && \
microdnf clean all
RUN --mount=type=cache,target=/var/cache/yum \
microdnf install -y \
cuda-nvcc-12-4 \
cuda-nvtx-12-4 \
cuda-cudart-12-4 \
cuda-compat-12-4 \
cuda-libraries-devel-12-4 \
&& microdnf clean all

ENV CUDA_HOME="/usr/local/cuda" \
PATH="${CUDA_HOME}/bin:${PATH}" \
Expand All @@ -57,23 +67,25 @@ ENV PATH="$VIRTUAL_ENV/bin:$PATH"

# install cuda and common dependencies
RUN --mount=type=cache,target=/root/.cache/pip \
--mount=type=cache,target=/root/.cache/uv \
--mount=type=bind,source=requirements-common.txt,target=requirements-common.txt \
--mount=type=bind,source=requirements-cuda.txt,target=requirements-cuda.txt \
pip install \
uv pip install \
-r requirements-cuda.txt

## Development #################################################################
FROM python-cuda-base AS dev

# install build and runtime dependencies
RUN --mount=type=cache,target=/root/.cache/pip \
--mount=type=cache,target=/root/.cache/uv \
--mount=type=bind,source=requirements-common.txt,target=requirements-common.txt \
--mount=type=bind,source=requirements-cuda.txt,target=requirements-cuda.txt \
--mount=type=bind,source=requirements-dev.txt,target=requirements-dev.txt \
--mount=type=bind,source=requirements-lint.txt,target=requirements-lint.txt \
--mount=type=bind,source=requirements-adag.txt,target=requirements-adag.txt \
--mount=type=bind,source=requirements-test.txt,target=requirements-test.txt \
pip3 install \
uv pip install \
-r requirements-cuda.txt \
-r requirements-dev.txt

Expand All @@ -82,12 +94,17 @@ FROM dev AS build

# install build dependencies
RUN --mount=type=cache,target=/root/.cache/pip \
--mount=type=cache,target=/root/.cache/uv \
--mount=type=bind,source=requirements-build.txt,target=requirements-build.txt \
pip install -r requirements-build.txt
uv pip install -r requirements-build.txt

# install compiler cache to speed up compilation leveraging local or remote caching
# git is required for the cutlass kernels
RUN rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && rpm -ql epel-release && microdnf install -y git ccache && microdnf clean all
RUN rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && \
rpm -ql epel-release && \
microdnf install -y git ccache && \
microdnf clean all

# install build dependencies

# copy input files
Expand Down Expand Up @@ -121,6 +138,7 @@ COPY vllm vllm
ENV CCACHE_DIR=/root/.cache/ccache
RUN --mount=type=cache,target=/root/.cache/ccache \
--mount=type=cache,target=/root/.cache/pip \
--mount=type=cache,target=/root/.cache/uv \
--mount=type=bind,src=.git,target=/workspace/.git \
env CFLAGS="-march=haswell" \
CXXFLAGS="$CFLAGS $CXXFLAGS" \
Expand All @@ -144,7 +162,7 @@ RUN curl -LO https://github.com/jedisct1/libsodium/releases/download/${LIBSODIUM
RUN ./configure --prefix="/usr/" && make && make check

## Release #####################################################################
FROM python-install AS vllm-openai
FROM python-cuda-base AS vllm-openai

WORKDIR /workspace

Expand All @@ -158,15 +176,17 @@ RUN microdnf install -y gcc \
# install vllm wheel first, so that torch etc will be installed
RUN --mount=type=bind,from=build,src=/workspace/dist,target=/workspace/dist \
--mount=type=cache,target=/root/.cache/pip \
pip install $(echo dist/*.whl)'[tensorizer]' --verbose
--mount=type=cache,target=/root/.cache/uv \
uv pip install $(echo dist/*.whl)'[tensorizer]' --verbose

# Install libsodium for Tensorizer encryption
RUN --mount=type=bind,from=libsodium-builder,src=/usr/src/libsodium,target=/usr/src/libsodium \
cd /usr/src/libsodium \
&& make install

RUN --mount=type=cache,target=/root/.cache/pip \
pip install https://github.com/flashinfer-ai/flashinfer/releases/download/v0.0.9/flashinfer-0.0.9+cu121torch2.3-cp311-cp311-linux_x86_64.whl
--mount=type=cache,target=/root/.cache/uv \
uv pip install https://github.com/flashinfer-ai/flashinfer/releases/download/v0.0.9/flashinfer-0.0.9+cu121torch2.3-cp311-cp311-linux_x86_64.whl

ENV HF_HUB_OFFLINE=1 \
PORT=8000 \
Expand All @@ -190,7 +210,7 @@ FROM vllm-openai as vllm-grpc-adapter
USER root

RUN --mount=type=cache,target=/root/.cache/pip \
pip install vllm-tgis-adapter==0.2.3
pip install vllm-tgis-adapter==0.2.4

ENV GRPC_PORT=8033
USER 2000
Expand Down
Loading