Skip to content

Commit

Permalink
Merge pull request #463 from buildkite-plugins/SUP-2585-drivers
Browse files Browse the repository at this point in the history
SUP-2585: Add Support for changing Build Drivers
  • Loading branch information
tomowatt authored Nov 21, 2024
2 parents 42fe6b5 + 8dd5ed5 commit 8a77104
Show file tree
Hide file tree
Showing 7 changed files with 566 additions and 3 deletions.
82 changes: 82 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -344,6 +344,88 @@ If set to true, docker compose will build with the `--with-dependencies` option

The default is `false`.

#### `builder` (object)

Defines the properties required for creating, using and removing Builder Instances. If not set, the default Builder Instance on the Agent Instance will be used.

##### `bootstrap` (boolean)

If set to true, will boot builder instance after creation. Optional when using `create`.

The default is `true`.

##### `create` (boolean)

If set to true, will use `docker buildx create` to create a new Builder Instance using the propeties defined. If a Builder Instance with the same `name` already exists, it will not be recreated.

The default is `false`.

##### `debug` (boolean)

If set to true, enables debug logging during creation of builder instance. Optional when using `create`.

The default is `false`.

##### `driver`

If set will create a Builder Instance using the selected Driver and use it. Available Drivers:

- `docker-container` creates a dedicated BuildKit container using Docker.
- `kubernetes` creates BuildKit pods in a Kubernetes cluster.
- `remote` connects directly to a manually managed BuildKit daemon.

More details on different [Build Drivers](https://docs.docker.com/build/builders/drivers/).

##### `driver-opt`

Commas separated, Key-Value pairs of driver-specific options to configure the Builder Instance when using `create`. Available options for each Driver:

- [docker-container](https://docs.docker.com/build/builders/drivers/docker-container/)
- [kubernetes](https://docs.docker.com/build/builders/drivers/kubernetes/)
- [remote](https://docs.docker.com/build/builders/drivers/remote/)

Example: `memory=100m,cpuset-cpus=1`

##### `keep-daemon` (boolean)

If set to true, will keep the BuildKit daemon running after the buildx context (Builder) is removed. This is useful when you manage BuildKit daemons and buildx contexts independently. Only supported by the `docker-container` and `kubernetes` drivers. Optional when using `remove`.

The default is `false`.

##### `keep-state` (boolean)

If set to true, will keep BuildKit state so it can be reused by a new Builder with the same name, persisting the driver's cache. Currently, only supported by the `docker-container` driver. Optional when using `remove`.

The default is `false`.

##### `name`

Sets the name of the Builder instance to create or use. Required when using `create` or `use` builder paramaters.

##### `platform`

Commas separated, fixed platforms for builder instance. Optional when using `create`.

Example: `linux/amd64,linux/arm64`

##### `remote-address`

Address of remote builder instance. Required when using `driver: remote`.

Example: `tcp://localhost:1234`

##### `remove` (boolean)

If set to true will stop and remove the Builder Instance specified by `name`.

The default is `false`.

##### `use` (boolean)

If set to true will use Builder Instance specified by `name`.

The default is `false`.

## Developing

To run the tests:
Expand Down
115 changes: 113 additions & 2 deletions docs/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -271,7 +271,7 @@ A newly spawned agent won't contain any of the docker caches for the first run w

```yaml
steps:
- label: ":docker Build an image"
- label: ":docker: Build an image"
plugins:
- docker-compose#v5.4.1:
build: app
Expand All @@ -295,7 +295,7 @@ The values you add in the `cache-from` will be mapped to the corresponding servi

```yaml
steps:
- label: ":docker Build an image"
- label: ":docker: Build an image"
plugins:
- docker-compose#v5.4.1:
build: app
Expand All @@ -312,3 +312,114 @@ steps:
push:
- app:myregistry:port/myrepo/myapp:latest
```

### Create, Use and Remove Builder Instances

#### Create

Most Docker setups, unless configured, will use the `docker` Builder Driver by default. More details on it [here](https://docs.docker.com/build/builders/drivers/docker/).

The `docker` driver can handle most situations but for advance features with the Docker, different Builder Drivers are required and this requires a Builder Instance being created first, which can be done manually or with the Plugin. To create a Builder Instance using a chosen Driver, requires the `name`, `driver` and `create` parameters:

```yaml
steps:
- label: ":docker: Build an image"
plugins:
- docker-compose#v5.4.1:
build: app
push: app:index.docker.io/myorg/myrepo:my-branch
cache-from:
- "app:myregistry:port/myrepo/myapp:my-branch"
- "app:myregistry:port/myrepo/myapp:latest"
builder:
name: container
driver: docker-container
create: true
```

**If a Builder Instance with the same `name` already exists, it will not be recreated.**

#### Use

By default, Builder Instances specified by `name` or that are created with `create` are not used, and the default Builder Instance on the Agent will be used. To use a Builder Instance, requires the `name` and `use` parameters and the Builder Instance to exist:

```yaml
steps:
- label: ":docker: Build an image"
plugins:
- docker-compose#v5.4.1:
build: app
push: app:index.docker.io/myorg/myrepo:my-branch
cache-from:
- "app:myregistry:port/myrepo/myapp:my-branch"
- "app:myregistry:port/myrepo/myapp:latest"
builder:
name: container
use: true
```

#### Remove

By default, Builder Instances specified by `name` or that are created with `create` are not removed after the Job finishs. To remove a Builder Instance, requires the `name` and `remove` parameters and the Builder Instance to exist:

```yaml
steps:
- label: ":docker: Build an image"
plugins:
- docker-compose#v5.4.1:
build: app
push: app:index.docker.io/myorg/myrepo:my-branch
cache-from:
- "app:myregistry:port/myrepo/myapp:my-branch"
- "app:myregistry:port/myrepo/myapp:latest"
builder:
name: container
driver: docker-container
create: true
use: true
remove: true
```

**Removing a Builder Instance by default will remove the daemon running it and its state (which can be used for cache).**
**To keep the daemon or state, use the `keep-daemon` or `keep-state` parameters.**
**These parameter are only applicable with specific Drivers, for detail see [`docker buildx rm`](https://docs.docker.com/reference/cli/docker/buildx/rm/).**

### Reusing caches from remote registries

A newly spawned agent won't contain any of the docker caches for the first run which will result in a long build step. To mitigate this you can reuse caches from a remote registry, but requires pushing cache and manifests to a registry using a Builder Driver that supports cache exports e.g., `docker-container` - the `docker` driver does not support this by default. For any remote registry used that requires authenication, see [Authenticated registries](#authenticated-registries) for more details. This requires use of the `cache-from`, `cache-to`, `name` and `use` parameters but will use the `create` and `driver` parameters to create the Builder Instance across multiple Agents:

```yaml
steps:
- label: ":docker: Build an image and push cache"
plugins:
- docker-compose#v5.4.1:
build: app
push: app:${DOCKER_REGISTRY}/${IMAGE_REPO}:cache
cache-from:
- "app:type=registry,ref=${DOCKER_REGISTRY}/${IMAGE_REPO}:cache"
cache-to:
- "app:type=registry,mode=max,image-manifest=true,oci-mediatypes=true,ref=${DOCKER_REGISTRY}/${IMAGE_REPO}:cache"
builder:
name: container
use: true
create: true
driver: docker-container
- wait
- label: ":docker: Build an image using remote cache"
plugins:
- docker-compose#v5.4.1:
build: app
cache-from:
- "app:type=registry,ref=${DOCKER_REGISTRY}/${IMAGE_REPO}:cache"
builder:
name: container
use: true
create: true
driver: docker-container
```

The first Step will build the Image using a Builder Instance with the `docker-container` driver and push the image cache to the remote registry, as specified by `cache-to`, with additional cache export options being used to export all the layers of intermediate steps with the image manifests. More details cache export options [here](https://github.com/moby/buildkit?tab=readme-ov-file#registry-push-image-and-cache-separately).

The second Step will build the Image using a Builder Instance with the `docker-container` driver and use remote registry for the image cache, as specified by `cache-from`, speeding up Image building process.
85 changes: 85 additions & 0 deletions hooks/pre-command
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
#!/bin/bash
set -ueo pipefail

DIR="$(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)"

# shellcheck source=lib/shared.bash
. "$DIR/../lib/shared.bash"

builder_create="$(plugin_read_config BUILDER_CREATE "false")"
builder_use="$(plugin_read_config BUILDER_USE "false")"
builder_name="$(plugin_read_config BUILDER_NAME "")"

if [[ "${builder_create}" == "true" ]] || [[ "${builder_use}" == "true" ]]; then
if [[ -z "${builder_name}" ]]; then
echo "+++ 🚨 Builder Name cannot be empty when using 'create' or 'use' parameters"
exit 1
fi
fi

if [[ "${builder_create}" == "true" ]]; then
if ! builder_instance_exists "${builder_name}"; then
builder_instance_args=(--name "${builder_name}")

build_driver="$(plugin_read_config BUILDER_DRIVER "")"
valid_drivers="docker-container|kubernetes|remote"
if [[ "${build_driver}" =~ ^(${valid_drivers})$ ]]; then
builder_instance_args+=(--driver "${build_driver}")
else
echo "+++ 🚨 Invalid driver: '${build_driver}'"
echo "Valid Drivers: ${valid_drivers//|/, }"
exit 1
fi

if [[ "$(plugin_read_config BUILDER_BOOTSTRAP "true")" == "true" ]]; then
builder_instance_args+=("--bootstrap")
fi

if [[ "$(plugin_read_config BUILDER_DEBUG "false")" == "true" ]]; then
builder_instance_args+=("--debug")
fi

driver_opt="$(plugin_read_config BUILDER_DRIVER_OPT "")"
if [[ -n "${driver_opt}" ]]; then
builder_instance_args+=("--driver-opt ${driver_opt}")
fi

builder_platform="$(plugin_read_config BUILDER_PLATFORM "")"
if [[ -n "${builder_platform}" ]]; then
builder_instance_args+=("--platform ${builder_platform}")
fi

remote_address="$(plugin_read_config BUILDER_REMOTE_ADDRESS "")"
if [[ -n "${remote_address}" ]]; then
if [[ "${build_driver}" == "remote" ]]; then
builder_instance_args+=("${remote_address}")
else
echo "+++ 🚨 Builder Driver '${build_driver}' used with 'remote-address' parameter"
echo "'remote-address' can only be used with 'remote' driver"
exit 1
fi
fi

echo "~~~ :docker: Creating Builder Instance '${builder_name}' with Driver '${build_driver}'"
docker buildx create "${builder_instance_args[@]}"
if [[ "${builder_use}" == "false" ]]; then
echo "~~~ :warning: Builder Instance '${builder_name}' created but will not be used as 'use: true' parameter not specified"
fi
else
echo "~~~ :docker: Not Creating Builder Instance '${builder_name}' as already exists"
fi
fi

if [[ "${builder_use}" == "true" ]]; then
if builder_instance_exists "${builder_name}"; then
echo "~~~ :docker: Using Builder Instance '${builder_name}'"
docker buildx use "${builder_name}"
else
echo "+++ 🚨 Builder Instance '${builder_name}' does not exist"
exit 1
fi
else
current_builder=$(docker buildx inspect | grep -m 1 "Name" | awk '{print $2}') || true
current_builder_driver=$(docker buildx inspect | grep -m 1 "Driver" | awk '{print $2}') || true
echo "~~~ :docker: Using Default Builder '${current_builder}' with Driver '${current_builder_driver}'"
fi
27 changes: 26 additions & 1 deletion hooks/pre-exit
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,35 @@ rm -f "docker-compose.buildkite-${BUILDKITE_BUILD_NUMBER}-override.yml"

# clean up resources after a run command. we do this here so that it will
# run after a job is cancelled
if [[ -n "$(plugin_read_list RUN)" ]] && [[ "$(plugin_read_config CLEANUP "true")" == "true" ]] ; then
if [[ -n "$(plugin_read_list RUN)" ]] && [[ "$(plugin_read_config CLEANUP "true")" == "true" ]]; then
# shellcheck source=lib/run.bash
. "$DIR/../lib/run.bash"

echo "~~~ :docker: Cleaning up after docker-compose" >&2
compose_cleanup
fi

# clean up builder instances if specified
builder_name="$(plugin_read_config BUILDER_NAME "")"
builder_remove="$(plugin_read_config BUILDER_REMOVE "false")"

if [[ "${builder_remove}" == true ]]; then
if builder_instance_exists "${builder_name}"; then

builder_remove_args=("${builder_name}")

if [[ "$(plugin_read_config BUILDER_KEEP_DAEMON "false")" == "true" ]]; then
builder_remove_args+=("--keep-daemon")
fi

if [[ "$(plugin_read_config BUILDER_KEEP_STATE "false")" == "true" ]]; then
builder_remove_args+=("--keep-state")
fi

echo "~~~ :docker: Cleaning up Builder Instance '${builder_name}'"
docker buildx stop "${builder_name}"
docker buildx rm "${builder_remove_args[@]}"
else
echo "~~~ :warning: Cannot remove Builder Instance '${builder_name}' as does not exist"
fi
fi
11 changes: 11 additions & 0 deletions lib/shared.bash
Original file line number Diff line number Diff line change
Expand Up @@ -320,3 +320,14 @@ function validate_tag {
return 1
fi
}

function builder_instance_exists() {
local builder_name="$1"

# Check if the specified builder exists by suppressing output and checking the exit status
if docker buildx inspect "${builder_name}" >/dev/null 2>&1; then
return 0 # Builder exists
else
return 1 # Builder does not exist
fi
}
29 changes: 29 additions & 0 deletions plugin.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,35 @@ configuration:
build-alias:
type: [ string, array ]
minimum: 1
builder:
type: object
properties:
bootstrap:
type: boolean
create:
type: boolean
debug:
type: boolean
driver:
type: string
enum: [ "docker-container", "kubernetes", "remote" ]
driver-opt:
type: string
keep-daemon:
type: boolean
keep-state:
type: boolean
name:
type: string
platform:
type: string
remote-address:
type: string
remove:
type: boolean
use:
type: boolean
additionalProperties: false
build-labels:
type: [ string, array ]
minimum: 1
Expand Down
Loading

0 comments on commit 8a77104

Please sign in to comment.