Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid (some) feature unification in CI builds #2636

Merged
merged 2 commits into from
Feb 6, 2025
Merged

Avoid (some) feature unification in CI builds #2636

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dc51320
Avoid (some) feature unification in CI builds
jackkleeman Feb 5, 2025
f0e041e
Actually set buildIndividually
jackkleeman Feb 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,14 @@ jobs:
with:
cache-on-failure: "true"

- name: Install cargo-hakari
uses: taiki-e/install-action@v2
with:
tool: cargo-hakari

- name: Disable hakari
run: cargo hakari disable
Comment on lines +65 to +71
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you observe an impact of this on CI builds (without the other bits)? Just curious to know the delta.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's an impact, and I'm not sure I'd expect one as I believe that our test will be doing feature unification anyway?


- name: Install protoc
uses: ./.github/actions/install-protoc

Expand Down
1 change: 1 addition & 0 deletions .github/workflows/docker-release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ jobs:
with:
pushToDockerHub: true
parca: true
buildIndividually: true

docker-cli:
name: Push CLI Docker image
Expand Down
57 changes: 38 additions & 19 deletions .github/workflows/docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,67 +4,77 @@ on:
workflow_call:
inputs:
uploadImageAsTarball:
description: 'uploads the Docker image additionally as a tarball'
description: "uploads the Docker image additionally as a tarball"
required: false
default: false
type: boolean
platforms:
description: 'platforms for docker build step'
description: "platforms for docker build step"
required: false
default: ''
default: ""
type: string
debug:
description: 'include debug symbols in built image'
description: "include debug symbols in built image"
required: false
default: false
type: boolean
parca:
description: 'split out debug symbols and upload to polar signals'
description: "split out debug symbols and upload to polar signals"
required: false
default: false
type: boolean
buildIndividually:
description: "build separate binaries individually to avoid feature unification"
required: false
default: false
type: boolean
features:
description: 'features to enable in the build'
description: "features to enable in the build"
required: false
default: ''
default: ""
type: string
pushToDockerHub:
description: 'push image to DockerHub'
description: "push image to DockerHub"
required: false
default: false
type: boolean

workflow_dispatch:
inputs:
uploadImageAsTarball:
description: 'uploads the Docker image additionally as a tarball'
description: "uploads the Docker image additionally as a tarball"
required: false
default: false
type: boolean
platforms:
description: 'platforms for docker build step'
description: "platforms for docker build step"
required: false
default: ''
default: ""
type: string
debug:
description: 'include debug symbols in built image'
description: "include debug symbols in built image"
required: false
default: false
type: boolean
parca:
description: 'upload debug symbols to polar signals'
description: "upload debug symbols to polar signals"
required: false
default: false
type: boolean
buildIndividually:
description: "build separate binaries individually to avoid feature unification"
required: false
default: false
type: boolean
pushToDockerHub:
description: 'push image to DockerHub'
description: "push image to DockerHub"
required: false
default: false
type: boolean

env:
REPOSITORY_OWNER: ${{ github.repository_owner }}
GHCR_REGISTRY: 'ghcr.io'
GHCR_REGISTRY: "ghcr.io"
GHCR_REGISTRY_USERNAME: ${{ github.actor }}
GHCR_REGISTRY_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Expand All @@ -77,6 +87,14 @@ jobs:
- name: Checkout repository
uses: actions/checkout@v4

- name: Install cargo-hakari
uses: taiki-e/install-action@v2
with:
tool: cargo-hakari

- name: Disable hakari
run: cargo hakari disable

# this is needed to be able to load and push a multiplatform image in one step
- name: Set up Docker containerd snapshotter
uses: crazy-max/ghaction-setup-docker@v3
Expand All @@ -97,7 +115,7 @@ jobs:
# https://docs.warpbuild.com/cache/docker-layer-caching#step-1-set-up-docker-buildx-action
driver-opts: |
network=host

- name: Cache sccache
id: cache
uses: WarpBuilds/cache@v1
Expand All @@ -113,7 +131,7 @@ jobs:
"sccache-cache": "/var/cache/sccache"
}
skip-extraction: ${{ steps.cache.outputs.cache-hit }}

- name: Log into GitHub container registry
uses: docker/login-action@v3
with:
Expand All @@ -137,7 +155,7 @@ jobs:
id: meta
uses: docker/metadata-action@v5
with:
images: |
images: |
${{ env.GHCR_REGISTRY }}/${{ env.REPOSITORY_OWNER }}/${{ env.IMAGE_NAME }}
${{ inputs.pushToDockerHub && format('docker.io/{0}/{1}', env.REPOSITORY_OWNER, env.IMAGE_NAME) || '' }}

Expand Down Expand Up @@ -166,13 +184,14 @@ jobs:
network: host
build-args: |
CARGO_PROFILE_RELEASE_DEBUG=${{ inputs.debug || inputs.parca }}
BUILD_INDIVIDUALLY=${{ inputs.buildIndividually }}
UPLOAD_DEBUGINFO=${{ inputs.parca }}
RESTATE_FEATURES=${{ inputs.features || '' }}
secrets: |
parca=${{ secrets.PARCA_TOKEN }}
cache-from: type=gha,url=http://127.0.0.1:49160/
cache-to: type=gha,url=http://127.0.0.1:49160/,mode=max

- name: Upload docker image tar as artifact
if: ${{ inputs.uploadImageAsTarball }}
uses: actions/upload-artifact@v4
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,12 @@ jobs:
if: "${{ runner.os == 'macOS' }}"
env:
"GITHUB_TOKEN": "${{ secrets.GITHUB_TOKEN }}"
- name: "Install cargo-hakari"
uses: "taiki-e/install-action@v2"
with:
"tool": "cargo-hakari"
- name: "Disable hakari"
run: "cargo hakari disable"
- name: "Allowlist git directory"
if: "${{ runner.os == 'Linux' }}"
run: "git config --global --add safe.directory /__w/restate/restate"
Expand Down
8 changes: 8 additions & 0 deletions .github/workflows/steps/release-build-setup.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,14 @@
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Install cargo-hakari
uses: taiki-e/install-action@v2
with:
tool: cargo-hakari

- name: Disable hakari
run: cargo hakari disable

- name: Allowlist git directory
if: ${{ runner.os == 'Linux' }} # only required inside the dev-tools container
run: git config --global --add safe.directory /__w/restate/restate
Expand Down
18 changes: 16 additions & 2 deletions docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,28 @@ ENV SCCACHE_DIR=/var/cache/sccache
# Overrides the behaviour of the release profile re including debug symbols, which in our repo is not to include them.
# Should be set to 'false' or 'true'. See https://doc.rust-lang.org/cargo/reference/environment-variables.html
ARG CARGO_PROFILE_RELEASE_DEBUG=false
# Avoids feature unification by building the three binaries individually
ARG BUILD_INDIVIDUALLY=false
ARG RESTATE_FEATURES=''
RUN just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES chef-cook --release --bins
RUN if [ "$BUILD_INDIVIDUALLY" = "true" ]; then \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES chef-cook --release -p restate-cli && \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES chef-cook --release -p restate-server && \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES chef-cook --release -p restatectl; \
else \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES chef-cook --release -p restate-cli -p restate-server -p restatectl; \
fi
COPY . .
# Mount the sccache directory as a cache to leverage sccache during build
# Caching layer if nothing has changed
# Use sccache during the main build
RUN --mount=type=cache,target=/var/cache/sccache \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES build --release --bins && \
if [ "$BUILD_INDIVIDUALLY" = "true" ]; then \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES build --release -p restate-cli && \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES build --release -p restate-server && \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES build --release -p restatectl; \
else \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES build --release -p restate-cli -p restate-server -p restatectl; \
fi && \
just notice-file && \
mv target/$(just arch=$TARGETARCH libc=gnu print-target)/release/restate-server target/restate-server && \
mv target/$(just arch=$TARGETARCH libc=gnu print-target)/release/restatectl target/restatectl && \
Expand Down
21 changes: 16 additions & 5 deletions docker/debug.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,17 +30,28 @@ ARG TARGETARCH
ENV RUSTC_WRAPPER=/usr/bin/sccache
ENV SCCACHE_DIR=/var/cache/sccache

# Overrides the behaviour of the release profile re including debug symbols, which in our repo is not to include them.
# Should be set to 'false' or 'true'. See https://doc.rust-lang.org/cargo/reference/environment-variables.html
ARG CARGO_PROFILE_RELEASE_DEBUG=false
# Avoids feature unification by building the three binaries individually
ARG BUILD_INDIVIDUALLY=false
ARG RESTATE_FEATURES=''
RUN just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES chef-cook --bins
RUN if [ "$BUILD_INDIVIDUALLY" = "true" ]; then \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES chef-cook -p restate-cli && \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES chef-cook -p restate-server && \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES chef-cook -p restatectl; \
else \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES chef-cook -p restate-cli -p restate-server -p restatectl; \
fi
COPY . .
# Mount the sccache directory as a cache to leverage sccache during build
# Caching layer if nothing has changed
# Use sccache during the main build
RUN --mount=type=cache,target=/var/cache/sccache \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES build --bins && \
if [ "$BUILD_INDIVIDUALLY" = "true" ]; then \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES build -p restate-cli && \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES build -p restate-server && \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES build -p restatectl; \
else \
just arch=$TARGETARCH libc=gnu features=$RESTATE_FEATURES build -p restate-cli -p restate-server -p restatectl; \
fi && \
just notice-file && \
mv target/$(just arch=$TARGETARCH libc=gnu print-target)/debug/restate-server target/restate-server && \
mv target/$(just arch=$TARGETARCH libc=gnu print-target)/debug/restatectl target/restatectl && \
Expand Down
Loading