From 4f73fe4721644103b53e098a68b2bc60240c0bf1 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 30 Jun 2021 17:43:21 -0400 Subject: [PATCH 01/45] Add RFC: Support Dockerfiles Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 113 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 text/0000-dockerfiles.md diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md new file mode 100644 index 000000000..ebe9cf409 --- /dev/null +++ b/text/0000-dockerfiles.md @@ -0,0 +1,113 @@ +# Meta +[meta]: #meta +- Name: Support Dockerfiles +- Start Date: 2021-06-30 +- Author(s): sclevine +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: [RFC0069](https://github.com/buildpacks/rfcs/blob/main/text/0069-stack-buildpacks.md), [RFC#167](https://github.com/buildpacks/rfcs/pull/167) +- Depends on: [RFC#172](https://github.com/buildpacks/rfcs/pull/172) + +# Summary +[summary]: #summary + +This RFC introduces functionality for customizing base images, as an alternative to stackpacks. + +# Motivation +[motivation]: #motivation + +Relying on Dockerfiles for base image generation and manipulation allows us to apply buildpacks only to the problem that they solve best: managing application runtimes and dependencies. + +# What it is +[what-it-is]: #what-it-is + +This RFC proposes that we replace stackpacks with multi-purpose build-time and runtime Dockerfiles. + +# How it Works +[how-it-works]: #how-it-works + +Note: kaniko, BuildKit, and/or the original Docker daemon may be used to apply Dockerfiles at the platform's discretion. + +### App-specified Dockerfiles + +A buildpack app may have a build.Dockerfile and/or run.Dockerfile in its app directory. A run.Dockerfile is applied to the selected runtime base image after the detection phase. A build.Dockerfile is applied to the build-time base image before the detection phase. + +Both Dockerfiles must accept `base_image` and `build_id` args. The `base_image` arg allows the lifecycle to specify the original base image. The `build_id` arg allows the app developer to bust the cache after a certain layer and must be defaulted to `0`. + +A runtime base image may indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. In the case of app-specified Dockerfiles, `io.buildpacks.rebasable=false` is set automatically before `run.Dockerfile` is applied and must be explicitly set to `true` if desired. Rebasing an app without this label set to `true` requires passing a new `--force` flag to `pack rebase`. + +### Platform-specified Dockerfiles + +The same Dockerfiles may be used to create new stacks or modify existing stacks outside of the app build process. For both app-specified and stack-modifying Dockerfiles, any specified labels override existing values. + +Dockerfiles that are used to create a stack must create a `/cnb/stack/genpkgs` executable that outputs a [CycloneDX](https://cyclonedx.org)-formatted list of packages in the image with PURL IDs when invoked. This executable is executed after any run.Dockerfile or build.Dockerfile is applied, and the output replaces the label `io.buildpacks.sbom`. This label doubles as a Software Bill-of-Materials for the base image. In the future, this label will serve as a starting point for the application SBoM. + +### Examples + +run.Dockerfile used to create a runtime base image: + +``` +ARG base_image +FROM ${base_image} +ARG build_id=0 + +LABEL io.buildpacks.image.distro=ubuntu +LABEL io.buildpacks.image.version=18.04 +LABEL io.buildpacks.rebasable=true + +ENV CNB_USER_ID=1234 +ENV CNB_GROUP_ID=1235 + +RUN groupadd cnb --gid ${CNB_GROUP_ID} && \ + useradd --uid ${CNB_USER_ID} --gid ${CNB_GROUP_ID} -m -s /bin/bash cnb + +USER ${CNB_USER_ID}:${CNB_GROUP_ID} + +COPY genpkgs /cnb/stack/genpkgs +``` + +run.Dockerfile present in an app directory that always installs the latest version of curl: +``` +ARG base_image +FROM ${base_image} +ARG build_id=0 + +LABEL io.buildpacks.rebasable=true + +RUN echo ${build_id} + +RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/* +``` + +Unsafe run.Dockerfile present in an app directory: +``` +ARG base_image +FROM ${base_image} +ARG build_id=0 + +LABEL io.buildpacks.rebasable=false + +RUN curl -L https://example.com/mypkg-install | sh # installs to /opt +``` + +# Drawbacks +[drawbacks]: #drawbacks + +- Involves breaking changes. +- Buildpacks cannot install OS packages directly, only select runtime base images. + +# Alternatives +[alternatives]: #alternatives + +- Stackpacks + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- Should `genpkgs` be part of this proposal? Opinion: Yes, otherwise it's difficult to maintain a valid SBoM. + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +This RFC requires extensive changes to all specifications. From ef2b23012fc53ed0eb1f037250167655520ae53a Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Thu, 1 Jul 2021 00:02:35 -0400 Subject: [PATCH 02/45] RFC: Support Dockerfiles: remove stack references Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index ebe9cf409..d775abc40 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -27,7 +27,7 @@ This RFC proposes that we replace stackpacks with multi-purpose build-time and r # How it Works [how-it-works]: #how-it-works -Note: kaniko, BuildKit, and/or the original Docker daemon may be used to apply Dockerfiles at the platform's discretion. +Note: kaniko, buildah, BuildKit, the original Docker daemon may be used to apply Dockerfiles at the platform's discretion. ### App-specified Dockerfiles @@ -39,9 +39,9 @@ A runtime base image may indicate that it preserves ABI compatibility by adding ### Platform-specified Dockerfiles -The same Dockerfiles may be used to create new stacks or modify existing stacks outside of the app build process. For both app-specified and stack-modifying Dockerfiles, any specified labels override existing values. +The same Dockerfiles may be used to create new base images or modify existing base images outside of the app build process. For both app-specified and base-image-modifying Dockerfiles, any specified labels override existing values. -Dockerfiles that are used to create a stack must create a `/cnb/stack/genpkgs` executable that outputs a [CycloneDX](https://cyclonedx.org)-formatted list of packages in the image with PURL IDs when invoked. This executable is executed after any run.Dockerfile or build.Dockerfile is applied, and the output replaces the label `io.buildpacks.sbom`. This label doubles as a Software Bill-of-Materials for the base image. In the future, this label will serve as a starting point for the application SBoM. +Dockerfiles that are used to create a base image must create a `/cnb/image/genpkgs` executable that outputs a [CycloneDX](https://cyclonedx.org)-formatted list of packages in the image with PURL IDs when invoked. This executable is executed after any run.Dockerfile or build.Dockerfile is applied, and the output replaces the label `io.buildpacks.sbom`. This label doubles as a Software Bill-of-Materials for the base image. In the future, this label will serve as a starting point for the application SBoM. ### Examples @@ -64,7 +64,7 @@ RUN groupadd cnb --gid ${CNB_GROUP_ID} && \ USER ${CNB_USER_ID}:${CNB_GROUP_ID} -COPY genpkgs /cnb/stack/genpkgs +COPY genpkgs /cnb/image/genpkgs ``` run.Dockerfile present in an app directory that always installs the latest version of curl: From e21b2af2ddf2a8864626d9e1e2a6a88fce209524 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 14 Jul 2021 23:38:21 -0400 Subject: [PATCH 03/45] RFC: Support Dockerfiles: add builder-specified Dockerfiles Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 68 +++++++++++++++++++++++++++++++--------- 1 file changed, 54 insertions(+), 14 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index d775abc40..fa231c167 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -27,25 +27,65 @@ This RFC proposes that we replace stackpacks with multi-purpose build-time and r # How it Works [how-it-works]: #how-it-works -Note: kaniko, buildah, BuildKit, the original Docker daemon may be used to apply Dockerfiles at the platform's discretion. +Note: kaniko, buildah, BuildKit, or the original Docker daemon may be used to apply Dockerfiles at the platform's discretion. -### App-specified Dockerfiles +### Builder-specified Dockerfiles -A buildpack app may have a build.Dockerfile and/or run.Dockerfile in its app directory. A run.Dockerfile is applied to the selected runtime base image after the detection phase. A build.Dockerfile is applied to the build-time base image before the detection phase. +A builder may specify any number of executable "hooks" in `/cnb/hooks.d/`. +Hook files in this directory are executed in the context of app directory and must output Dockerfiles that are applied to the build-time and runtime base images before an image is built. -Both Dockerfiles must accept `base_image` and `build_id` args. The `base_image` arg allows the lifecycle to specify the original base image. The `build_id` arg allows the app developer to bust the cache after a certain layer and must be defaulted to `0`. +If a hook exits with a non-zero status value, the build fails. If a hook exits with a zero status value and no output, the hook is ignored. Directories and non-executable files are ignored. + +Each executable must be in the format `/cnb/hooks.d/.(build.|run.|)`, where build, run, or empty specify the build-time base image, runtime base image, or both bash images, respectively. The only valid format is `Dockerfile`, although support for, e.g. LLB JSON, could be added in the future. + +A runtime Dockerfile is applied to the selected runtime base image after the detection phase. +A build-time Dockerfile is applied to the build-time base image before the detection phase. + +Both Dockerfiles must accept `base_image` and `build_id` args. +The `base_image` arg allows the lifecycle to specify the original base image. +The `build_id` arg allows the app developer to bust the cache after a certain layer and must be defaulted to `0`. When the `$build_id` arg is referenced in a `RUN` instruction, all subsequent layerrs will be rebuilt on the next build (as the value will change). A runtime base image may indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. In the case of app-specified Dockerfiles, `io.buildpacks.rebasable=false` is set automatically before `run.Dockerfile` is applied and must be explicitly set to `true` if desired. Rebasing an app without this label set to `true` requires passing a new `--force` flag to `pack rebase`. + +#### Example: App-specified Dockerfile Hook + +This example hook would allow an app to provide runtime and build-time base image extensions as "run.Dockerfile" and "build.Dockerfile." +The app developer can decide whether the extensions are rebasable. + +##### `/cnb/hooks.d/app.build.Dockerfile` +``` +#!/bin/sh +cat build.Dockerfile +``` +##### `/cnb/hooks.d/app.run.Dockerfile` +``` +#!/bin/sh +cat run.Dockerfile +``` + +#### Example: RPM Dockerfile Hook + +This example hook would allow a builder to install RPMs for each language runtime. + +Note: The Dockerfiles referenced must disable rebasing, and build times will be slower compared to buildpack-provided runtimes. + +##### `/cnb/hooks.d/app.Dockerfile` +``` +#!/bin/sh +[[ -f Gemfile.lock ]] && cat /cnb/hooks.d/app.Dockerfile.d/Dockerfile-ruby +[[ -f package.json ]] && cat /cnb/hooks.d/app.Dockerfile.d/Dockerfile-node +``` + ### Platform-specified Dockerfiles -The same Dockerfiles may be used to create new base images or modify existing base images outside of the app build process. For both app-specified and base-image-modifying Dockerfiles, any specified labels override existing values. +The same Dockerfile format may be used to create new base images or modify existing base images outside of the app build process. Any specified labels override existing values. -Dockerfiles that are used to create a base image must create a `/cnb/image/genpkgs` executable that outputs a [CycloneDX](https://cyclonedx.org)-formatted list of packages in the image with PURL IDs when invoked. This executable is executed after any run.Dockerfile or build.Dockerfile is applied, and the output replaces the label `io.buildpacks.sbom`. This label doubles as a Software Bill-of-Materials for the base image. In the future, this label will serve as a starting point for the application SBoM. +Dockerfiles that are used to create a base image must create a `/cnb/image/genpkgs` executable that outputs a [CycloneDX](https://cyclonedx.org)-formatted list of packages in the image with PURL IDs when invoked. This executable is executed after all Dockerfiles are applied, and the output replaces the label `io.buildpacks.sbom`. This label doubles as a Software Bill-of-Materials for the base image. In the future, this label will serve as a starting point for the application SBoM. -### Examples +### Example Dockerfiles -run.Dockerfile used to create a runtime base image: +Dockerfile used to create a runtime base image: ``` ARG base_image @@ -67,35 +107,35 @@ USER ${CNB_USER_ID}:${CNB_GROUP_ID} COPY genpkgs /cnb/image/genpkgs ``` -run.Dockerfile present in an app directory that always installs the latest version of curl: +run.Dockerfile for use with the example `app.Dockerfile` hook that always installs the latest version of curl: ``` ARG base_image FROM ${base_image} ARG build_id=0 -LABEL io.buildpacks.rebasable=true - RUN echo ${build_id} RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/* ``` +(note: this Dockerfile disables rebasing as OS package installation is not rebasable) -Unsafe run.Dockerfile present in an app directory: +run.Dockerfile for use with the example `app.Dockerfile` hook that installs a special package to /opt: ``` ARG base_image FROM ${base_image} ARG build_id=0 -LABEL io.buildpacks.rebasable=false +LABEL io.buildpacks.rebasable=true RUN curl -L https://example.com/mypkg-install | sh # installs to /opt ``` +(note: rebasing is explicitly allowed because only a single directory in /opt is created) + # Drawbacks [drawbacks]: #drawbacks - Involves breaking changes. -- Buildpacks cannot install OS packages directly, only select runtime base images. # Alternatives [alternatives]: #alternatives From a1810ca08075da4c010df10679020f5d4131d8b4 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Thu, 15 Jul 2021 15:45:52 -0400 Subject: [PATCH 04/45] RFC: Support Dockerfiles: Fix reference to app-specified Dockerfiles Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index fa231c167..1ab34a336 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -45,7 +45,7 @@ Both Dockerfiles must accept `base_image` and `build_id` args. The `base_image` arg allows the lifecycle to specify the original base image. The `build_id` arg allows the app developer to bust the cache after a certain layer and must be defaulted to `0`. When the `$build_id` arg is referenced in a `RUN` instruction, all subsequent layerrs will be rebuilt on the next build (as the value will change). -A runtime base image may indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. In the case of app-specified Dockerfiles, `io.buildpacks.rebasable=false` is set automatically before `run.Dockerfile` is applied and must be explicitly set to `true` if desired. Rebasing an app without this label set to `true` requires passing a new `--force` flag to `pack rebase`. +A runtime base image may indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. In the case of builder-specified Dockerfiles, `io.buildpacks.rebasable=false` is set automatically before `run.Dockerfile` is applied and must be explicitly set to `true` if desired. Rebasing an app without this label set to `true` requires passing a new `--force` flag to `pack rebase`. #### Example: App-specified Dockerfile Hook From 82cbe67922d0fe3650cfddaa721ef3c8581ff620 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Thu, 15 Jul 2021 15:50:34 -0400 Subject: [PATCH 05/45] RFC: Support Dockerfiles: Remove run.Dockerfile reference Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 1ab34a336..7ae7b248b 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -45,7 +45,7 @@ Both Dockerfiles must accept `base_image` and `build_id` args. The `base_image` arg allows the lifecycle to specify the original base image. The `build_id` arg allows the app developer to bust the cache after a certain layer and must be defaulted to `0`. When the `$build_id` arg is referenced in a `RUN` instruction, all subsequent layerrs will be rebuilt on the next build (as the value will change). -A runtime base image may indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. In the case of builder-specified Dockerfiles, `io.buildpacks.rebasable=false` is set automatically before `run.Dockerfile` is applied and must be explicitly set to `true` if desired. Rebasing an app without this label set to `true` requires passing a new `--force` flag to `pack rebase`. +A runtime base image may indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. In the case of builder-specified Dockerfiles, `io.buildpacks.rebasable=false` is set automatically before a runtime Dockerfile is applied and must be explicitly set to `true` if desired. Rebasing an app without this label set to `true` requires passing a new `--force` flag to `pack rebase`. #### Example: App-specified Dockerfile Hook From a9daed9cf8da3bf318114b429a51fcc33dac608a Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Sun, 18 Jul 2021 16:58:43 -0400 Subject: [PATCH 06/45] RFC: Support Dockerfiles: address more feedback Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 41 ++++++++++++++++++++++++++++------------ 1 file changed, 29 insertions(+), 12 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 7ae7b248b..0e10f0e5c 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -31,15 +31,32 @@ Note: kaniko, buildah, BuildKit, or the original Docker daemon may be used to ap ### Builder-specified Dockerfiles -A builder may specify any number of executable "hooks" in `/cnb/hooks.d/`. -Hook files in this directory are executed in the context of app directory and must output Dockerfiles that are applied to the build-time and runtime base images before an image is built. +A builder image may include any number of "hook" files in `/cnb/hooks.d/`. +The `pack create-builder` command is used to copy hooks into the builder image via `builder.toml`. E.g., +```toml +[[hooks]] +name = "app-hook" +path = "./myhook" +``` +(Note: this format should not be considered final. Additional fields will be required to mark the format, target, etc. of the hook. These may be defined in subsequent spec PRs / sub-team RFCs.) + +Each hook file path must be in the format `/cnb/hooks.d/.(build.|run.|)(.out|)`, where: -If a hook exits with a non-zero status value, the build fails. If a hook exits with a zero status value and no output, the hook is ignored. Directories and non-executable files are ignored. +1. `build`, `run`, or empty specify the build-time base image, runtime base image, or both bash images, respectively. +2. The only valid format is `Dockerfile`, although support for, e.g. LLB JSON, could be added in the future. +3. If the `.out` suffix is present and the file is executable, the hook is executed in the context of the app directory, and its output must match ``. +4. If the `.out` suffix is not present, the contents of the file must match ``. -Each executable must be in the format `/cnb/hooks.d/.(build.|run.|)`, where build, run, or empty specify the build-time base image, runtime base image, or both bash images, respectively. The only valid format is `Dockerfile`, although support for, e.g. LLB JSON, could be added in the future. +Hook files are evaluated (either read or executed) and the resulting Dockerfiles are applied to the build-time and runtime base images in lexographical order by ``. +Executable hook files must not modify the app directory when executed and may be executed in parallel. +However, the resulting Dockerfiles must not be applied in parallel. +A Dockerfile intended for the runtime base image is applied after the detection phase. +A Dockerfile intended for the build-time base image is applied before the detection phase. -A runtime Dockerfile is applied to the selected runtime base image after the detection phase. -A build-time Dockerfile is applied to the build-time base image before the detection phase. +If an executable hook exits with a non-zero status value, the build fails. +If a executable hook exits with a zero status value and no output, the hook is ignored. +Directories are ignored. +Files at the top-level of `/cnb/hooks.d/` that do not match the specified file name format result in a build failure. Both Dockerfiles must accept `base_image` and `build_id` args. The `base_image` arg allows the lifecycle to specify the original base image. @@ -53,12 +70,12 @@ A runtime base image may indicate that it preserves ABI compatibility by adding This example hook would allow an app to provide runtime and build-time base image extensions as "run.Dockerfile" and "build.Dockerfile." The app developer can decide whether the extensions are rebasable. -##### `/cnb/hooks.d/app.build.Dockerfile` +##### `/cnb/hooks.d/app.build.Dockerfile.out` ``` #!/bin/sh cat build.Dockerfile ``` -##### `/cnb/hooks.d/app.run.Dockerfile` +##### `/cnb/hooks.d/app.run.Dockerfile.out` ``` #!/bin/sh cat run.Dockerfile @@ -70,7 +87,7 @@ This example hook would allow a builder to install RPMs for each language runtim Note: The Dockerfiles referenced must disable rebasing, and build times will be slower compared to buildpack-provided runtimes. -##### `/cnb/hooks.d/app.Dockerfile` +##### `/cnb/hooks.d/app.Dockerfile.out` ``` #!/bin/sh [[ -f Gemfile.lock ]] && cat /cnb/hooks.d/app.Dockerfile.d/Dockerfile-ruby @@ -107,7 +124,7 @@ USER ${CNB_USER_ID}:${CNB_GROUP_ID} COPY genpkgs /cnb/image/genpkgs ``` -run.Dockerfile for use with the example `app.Dockerfile` hook that always installs the latest version of curl: +`run.Dockerfile` for use with the example `app.run.Dockerfile.out` hook that always installs the latest version of curl: ``` ARG base_image FROM ${base_image} @@ -117,9 +134,9 @@ RUN echo ${build_id} RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/* ``` -(note: this Dockerfile disables rebasing as OS package installation is not rebasable) +(note: this Dockerfile disables rebasing, as OS package installation is not rebasable) -run.Dockerfile for use with the example `app.Dockerfile` hook that installs a special package to /opt: +`run.Dockerfile` for use with the example `app.run.Dockerfile.out` hook that installs a special package to /opt: ``` ARG base_image FROM ${base_image} From 3534dc299a80d13151f03e94144369dfab9de29c Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Tue, 20 Jul 2021 15:51:51 -0400 Subject: [PATCH 07/45] RFC: Support Dockerfiles: fix typo Signed-off-by: Stephen Levine Co-authored-by: Daniel Mikusa --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 0e10f0e5c..d727c87ea 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -42,7 +42,7 @@ path = "./myhook" Each hook file path must be in the format `/cnb/hooks.d/.(build.|run.|)(.out|)`, where: -1. `build`, `run`, or empty specify the build-time base image, runtime base image, or both bash images, respectively. +1. `build`, `run`, or empty specify the build-time base image, runtime base image, or both base images, respectively. 2. The only valid format is `Dockerfile`, although support for, e.g. LLB JSON, could be added in the future. 3. If the `.out` suffix is present and the file is executable, the hook is executed in the context of the app directory, and its output must match ``. 4. If the `.out` suffix is not present, the contents of the file must match ``. From 05dc2d3de3de4470274b4b329c17ca5a08e3c092 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 21 Jul 2021 22:28:10 -0400 Subject: [PATCH 08/45] RFC: Support Dockerfiles: Buildpack-associated hooks Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 84 +++++++++++++++++++++++++++++++++------- 1 file changed, 71 insertions(+), 13 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index d727c87ea..bd2e0b3ca 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -22,22 +22,28 @@ Relying on Dockerfiles for base image generation and manipulation allows us to a # What it is [what-it-is]: #what-it-is -This RFC proposes that we replace stackpacks with multi-purpose build-time and runtime Dockerfiles. +This RFC proposes that we replace stackpacks with build-time and runtime Dockerfiles that act as pre-build hooks. +These hooks would be equivalent to base image / builder customizations, but would execute immediately before a build. + +Most importantly: the changes below do not impact buildpacks or the buildpack API. +For a given application, a build that uses hooks could be optimized by creating a more narrowly-scoped builder with the hooks pre-applied, without modifying the buildpacks. # How it Works [how-it-works]: #how-it-works Note: kaniko, buildah, BuildKit, or the original Docker daemon may be used to apply Dockerfiles at the platform's discretion. -### Builder-specified Dockerfiles +### Dynamically-applied Dockerfiles -A builder image may include any number of "hook" files in `/cnb/hooks.d/`. +A builder image may include any number of "hook" files in `/cnb/hooks.d/`. Hooks may be associated with zero or more buildpack IDs. The `pack create-builder` command is used to copy hooks into the builder image via `builder.toml`. E.g., ```toml [[hooks]] name = "app-hook" path = "./myhook" +buildpacks = ["com.example.buildpack1"] ``` + (Note: this format should not be considered final. Additional fields will be required to mark the format, target, etc. of the hook. These may be defined in subsequent spec PRs / sub-team RFCs.) Each hook file path must be in the format `/cnb/hooks.d/.(build.|run.|)(.out|)`, where: @@ -47,14 +53,19 @@ Each hook file path must be in the format `/cnb/hooks.d/.(build.|run.|)`. 4. If the `.out` suffix is not present, the contents of the file must match ``. -Hook files are evaluated (either read or executed) and the resulting Dockerfiles are applied to the build-time and runtime base images in lexographical order by ``. -Executable hook files must not modify the app directory when executed and may be executed in parallel. -However, the resulting Dockerfiles must not be applied in parallel. -A Dockerfile intended for the runtime base image is applied after the detection phase. -A Dockerfile intended for the build-time base image is applied before the detection phase. +Additionally, each hook file may be accompanied by an additional file of the same name, but with the suffix `.buildpacks`. +This file contains a newline-separated list of buildpack IDs. +If present, the hook is ignored unless one of the buildpacks in the list is selected during the buildpack detection phase. + +Hook files are evaluated (either read or executed) after the detection phase. +All Dockerfiles are applied to base images after the hooks are evaluated and before the buildpack build phase. +The resulting Dockerfiles are applied to the build-time and runtime base images in lexographical order by ``. + +Executable hook files must not modify the app directory when executed and may be evaluated in parallel. +However, the resulting Dockerfiles must not be applied in parallel to the same base image. If an executable hook exits with a non-zero status value, the build fails. -If a executable hook exits with a zero status value and no output, the hook is ignored. +If an executable hook exits with a zero status value and no output, the hook is ignored. Directories are ignored. Files at the top-level of `/cnb/hooks.d/` that do not match the specified file name format result in a build failure. @@ -81,9 +92,9 @@ cat build.Dockerfile cat run.Dockerfile ``` -#### Example: RPM Dockerfile Hook +#### Example: RPM Dockerfile Hook (app-based) -This example hook would allow a builder to install RPMs for each language runtime. +This example hook would allow a builder to install RPMs for each language runtime, based on the app directory. Note: The Dockerfiles referenced must disable rebasing, and build times will be slower compared to buildpack-provided runtimes. @@ -94,9 +105,43 @@ Note: The Dockerfiles referenced must disable rebasing, and build times will be [[ -f package.json ]] && cat /cnb/hooks.d/app.Dockerfile.d/Dockerfile-node ``` -### Platform-specified Dockerfiles +#### Example: RPM Dockerfile Hook (buildpack-based) + +This example hook would allow a builder to install RPMs for each language runtime, based on the available buildpacks. + +Note: The Dockerfiles referenced must disable rebasing, and build times will be slower compared to buildpack-provided runtimes. + -The same Dockerfile format may be used to create new base images or modify existing base images outside of the app build process. Any specified labels override existing values. +##### `/cnb/hooks.d/ruby.Dockerfile` +``` +... +LABEL io.buildpacks.rebasable=false # default, not needed +RUN yum install ruby +... +``` + +##### `/cnb/hooks.d/ruby.Dockerfile.buildpacks` +``` +com.example.ruby +``` + +##### `/cnb/hooks.d/node.Dockerfile` +``` +... +LABEL io.buildpacks.rebasable=false # default, not needed +RUN yum install nodejs +... +``` + +##### `/cnb/hooks.d/node.Dockerfile.buildpacks` +``` +com.example.node +``` + + +### Dockerfiles for Base Images + +The same Dockerfile format may be used to create new base images or modify existing base images outside of the app build process (e.g., before creating a builder). Any specified labels override existing values. Dockerfiles that are used to create a base image must create a `/cnb/image/genpkgs` executable that outputs a [CycloneDX](https://cyclonedx.org)-formatted list of packages in the image with PURL IDs when invoked. This executable is executed after all Dockerfiles are applied, and the output replaces the label `io.buildpacks.sbom`. This label doubles as a Software Bill-of-Materials for the base image. In the future, this label will serve as a starting point for the application SBoM. @@ -148,6 +193,17 @@ RUN curl -L https://example.com/mypkg-install | sh # installs to /opt ``` (note: rebasing is explicitly allowed because only a single directory in /opt is created) +### Future: Dynamic Run-time Base Image Selection + +The outcomes targeted by https://github.com/buildpacks/rfcs/pull/174 could be achieved with an additional type of hook. + +For example, we could introduce a `ref` format that replaces the image instead of extending it: +``` +/cnb/hooks.d/myhook.run.ref # contains: index.docker.io/example/my-ruby-run-image +/cnb/hooks.d/myhook.run.buildpacks # contains: com.example.ruby-buildpack +``` + +This would simplify buildpacks by removing the need for PURLs outside of SBoM generation. The `packages` table could be removed from both buildpack.toml and `/bin/detect`. # Drawbacks [drawbacks]: #drawbacks @@ -158,6 +214,8 @@ RUN curl -L https://example.com/mypkg-install | sh # installs to /opt [alternatives]: #alternatives - Stackpacks +- "Stackpacks-lite" -- Instead of `.buildpacks`, have hooks participate in buildpack detection (provide-only, must come first) +- Use directories for hooks: `/cnb/hooks.d/app.build.Dockerfile.out` -> `/cnb/hooks.d/app/build.Dockerfile.out` # Unresolved Questions [unresolved-questions]: #unresolved-questions From f3e84c6220421900fc4499333f0fd09cd36f0d2e Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Thu, 22 Jul 2021 21:23:35 -0400 Subject: [PATCH 09/45] RFC: Support Dockerfiles: clarify builder vs. build-image Signed-off-by: Stephen Levine Co-authored-by: Emily Casey --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index bd2e0b3ca..06819a814 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -48,7 +48,7 @@ buildpacks = ["com.example.buildpack1"] Each hook file path must be in the format `/cnb/hooks.d/.(build.|run.|)(.out|)`, where: -1. `build`, `run`, or empty specify the build-time base image, runtime base image, or both base images, respectively. +1. `build`, `run`, or empty specify the builder image, runtime base image, or both base images, respectively. 2. The only valid format is `Dockerfile`, although support for, e.g. LLB JSON, could be added in the future. 3. If the `.out` suffix is present and the file is executable, the hook is executed in the context of the app directory, and its output must match ``. 4. If the `.out` suffix is not present, the contents of the file must match ``. From 1e813b987ad6734169ed7c74ac60f51f943dd74f Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Thu, 29 Jul 2021 01:44:31 -0400 Subject: [PATCH 10/45] RFC: Support Dockerfiles: integrate with buildpack API Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 159 +++++++++++++++++---------------------- 1 file changed, 70 insertions(+), 89 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 06819a814..43e8fca07 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -12,21 +12,20 @@ # Summary [summary]: #summary -This RFC introduces functionality for customizing base images, as an alternative to stackpacks. +This RFC introduces functionality for customizing base images, as an alternative to stackpacks ([RFC0069](https://github.com/buildpacks/rfcs/blob/main/text/0069-stack-buildpacks.md)). # Motivation [motivation]: #motivation -Relying on Dockerfiles for base image generation and manipulation allows us to apply buildpacks only to the problem that they solve best: managing application runtimes and dependencies. +[RFC0069](https://github.com/buildpacks/rfcs/blob/main/text/0069-stack-buildpacks.md) introduces complexity by defining an API that allows buildpacks to modify base images. To avoid this complexity, we could rely on generated Dockerfiles for base image manipulation. This would simplify the original proposal by, e.g., only requiring a copy of the buildpack on the build-time base image. # What it is [what-it-is]: #what-it-is -This RFC proposes that we replace stackpacks with build-time and runtime Dockerfiles that act as pre-build hooks. -These hooks would be equivalent to base image / builder customizations, but would execute immediately before a build. +This RFC proposes that we replace stackpacks with dynamically-generated build-time and runtime Dockerfiles that act as pre-build hooks. +These hooks would participate in detection and execute before the buildpack build process. -Most importantly: the changes below do not impact buildpacks or the buildpack API. -For a given application, a build that uses hooks could be optimized by creating a more narrowly-scoped builder with the hooks pre-applied, without modifying the buildpacks. +For a given application, a build that uses hooks could be optimized by creating a more narrowly-scoped builder that does not contain hooks. # How it Works [how-it-works]: #how-it-works @@ -35,61 +34,88 @@ Note: kaniko, buildah, BuildKit, or the original Docker daemon may be used to ap ### Dynamically-applied Dockerfiles -A builder image may include any number of "hook" files in `/cnb/hooks.d/`. Hooks may be associated with zero or more buildpack IDs. -The `pack create-builder` command is used to copy hooks into the builder image via `builder.toml`. E.g., +A builder image may include any number of "hook" directories in `/cnb/hooks/`. + +Hooks are similar to buildpacks: they have a `/bin/build` and `/bin/detect` executable. +However, instead of a `buildpack.toml` file, hooks have a `hook.toml` file: ```toml -[[hooks]] -name = "app-hook" -path = "./myhook" -buildpacks = ["com.example.buildpack1"] +api = "" + +[hook] +id = "" +name = "" +version = "" +homepage = "" +description = "" +keywords = [ "" ] + +[[hook.licenses]] +type = "" +uri = "" ``` -(Note: this format should not be considered final. Additional fields will be required to mark the format, target, etc. of the hook. These may be defined in subsequent spec PRs / sub-team RFCs.) +Hooks may be packaged and examined similar to buildpacks, but with analogous `pack hook` subcommands. + +Other pack CLI commands, such as `pack builder create`, will be extended to include support for hooks. -Each hook file path must be in the format `/cnb/hooks.d/.(build.|run.|)(.out|)`, where: +Unlike buildpacks, +- Hooks must not be included in a meta-buildpacks +- Hooks must not have `order`/`group` definitions in `hook.toml` -1. `build`, `run`, or empty specify the builder image, runtime base image, or both base images, respectively. -2. The only valid format is `Dockerfile`, although support for, e.g. LLB JSON, could be added in the future. -3. If the `.out` suffix is present and the file is executable, the hook is executed in the context of the app directory, and its output must match ``. -4. If the `.out` suffix is not present, the contents of the file must match ``. +Hooks participate in the buildpack detection process, with the same interface for `/bin/detect`. +However, +- `/bin/detect` is optional for hooks, and they are assumed to pass detection when it is not presetn. +- Hooks may only output `provides` entries to the build plan. They must not output `requires`. +- Hooks must all proceed regular buildpacks in `order` definitions (e.g., in `builder.toml`). +- Hooks are always `optional`. -Additionally, each hook file may be accompanied by an additional file of the same name, but with the suffix `.buildpacks`. -This file contains a newline-separated list of buildpack IDs. -If present, the hook is ignored unless one of the buildpacks in the list is selected during the buildpack detection phase. +Hooks generate Dockerfiles before the regular buildpack build phase. +To generate these Dockerfiles, the lifecycle executes the hook's `/bin/build` executable with the same interface as regular buildpacks. +However, +- Hooks `/bin/build` must not write to the app directory. +- Hooks `/bin/build` may be executed in parallel. +- Hooks `` directory is replaced by an `` directory. +- If a hook is missing `/bin/build`, the hook root is treated as a pre-populated `` directory. -Hook files are evaluated (either read or executed) after the detection phase. -All Dockerfiles are applied to base images after the hooks are evaluated and before the buildpack build phase. -The resulting Dockerfiles are applied to the build-time and runtime base images in lexographical order by ``. +After `/bin/build` executes, the `` directory may contain +- `build.toml`, with the same contents as a buildpack's `build.toml` +- Either `Dockerfile` or either or both of `build.Dockerfile` and `run.Dockerfile` -Executable hook files must not modify the app directory when executed and may be evaluated in parallel. -However, the resulting Dockerfiles must not be applied in parallel to the same base image. +Support for other instruction formats, e.g., LLB JSON files, could be added in the future. -If an executable hook exits with a non-zero status value, the build fails. -If an executable hook exits with a zero status value and no output, the hook is ignored. -Directories are ignored. -Files at the top-level of `/cnb/hooks.d/` that do not match the specified file name format result in a build failure. +`build.Dockerfile`, `run.Dockerfile`, and `Dockerfile` target the builder image, runtime base image, or both base images, respectively. -Both Dockerfiles must accept `base_image` and `build_id` args. -The `base_image` arg allows the lifecycle to specify the original base image. -The `build_id` arg allows the app developer to bust the cache after a certain layer and must be defaulted to `0`. When the `$build_id` arg is referenced in a `RUN` instruction, all subsequent layerrs will be rebuilt on the next build (as the value will change). +If no Dockerfiles are present, `/bin/build` may still consume build plan entries and add metadata to `build.toml`. + +Dockerfiles are applied to their corresponding base images after all hooks are executed and before any regular buildpacks are executed. +Dockerfiles are applied in the order determined during buildpack detection. + +All Dockerfiles are provided with `base_image` and `build_id` args. +The `base_image` arg allows the Dockerfile to reference the original base image. +The `build_id` arg allows the Dockerfile to invalidate the cache after a certain layer and must be defaulted to `0`. +When the `$build_id` arg is referenced in a `RUN` instruction, all subsequent layerrs will be rebuilt on the next build (as the value will change). A runtime base image may indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. In the case of builder-specified Dockerfiles, `io.buildpacks.rebasable=false` is set automatically before a runtime Dockerfile is applied and must be explicitly set to `true` if desired. Rebasing an app without this label set to `true` requires passing a new `--force` flag to `pack rebase`. +Finally, base images may be statically labeled with any number of `provides` that are treated as build plan entries. +These `provides` may contain fields other than `name`, which, when mismatched with `requires`, mark the entry as `unmet` by the stack. +This is important to ensure that: +- Rebasing always remains an option for end users. +- Buildpacks do not become dependent on hooks. +- Builds can be time-optimized by creating base images ahead of time. + +NOTE: the above can be accomplished by a hook with a no-op `/bin/build` -- do we really need this? #### Example: App-specified Dockerfile Hook This example hook would allow an app to provide runtime and build-time base image extensions as "run.Dockerfile" and "build.Dockerfile." The app developer can decide whether the extensions are rebasable. -##### `/cnb/hooks.d/app.build.Dockerfile.out` -``` -#!/bin/sh -cat build.Dockerfile -``` -##### `/cnb/hooks.d/app.run.Dockerfile.out` +##### `/cnb/hooks/com.example.apphook/bin/build` ``` #!/bin/sh -cat run.Dockerfile +[ -f build.Dockerfile ] && cp build.Dockerfile "$1/" +[ -f run.Dockerfile ] && cp run.Dockerfile "$1/" ``` #### Example: RPM Dockerfile Hook (app-based) @@ -98,44 +124,11 @@ This example hook would allow a builder to install RPMs for each language runtim Note: The Dockerfiles referenced must disable rebasing, and build times will be slower compared to buildpack-provided runtimes. -##### `/cnb/hooks.d/app.Dockerfile.out` +##### `/cnb/hooks/com.example.rpmhook/bin/build` ``` #!/bin/sh -[[ -f Gemfile.lock ]] && cat /cnb/hooks.d/app.Dockerfile.d/Dockerfile-ruby -[[ -f package.json ]] && cat /cnb/hooks.d/app.Dockerfile.d/Dockerfile-node -``` - -#### Example: RPM Dockerfile Hook (buildpack-based) - -This example hook would allow a builder to install RPMs for each language runtime, based on the available buildpacks. - -Note: The Dockerfiles referenced must disable rebasing, and build times will be slower compared to buildpack-provided runtimes. - - -##### `/cnb/hooks.d/ruby.Dockerfile` -``` -... -LABEL io.buildpacks.rebasable=false # default, not needed -RUN yum install ruby -... -``` - -##### `/cnb/hooks.d/ruby.Dockerfile.buildpacks` -``` -com.example.ruby -``` - -##### `/cnb/hooks.d/node.Dockerfile` -``` -... -LABEL io.buildpacks.rebasable=false # default, not needed -RUN yum install nodejs -... -``` - -##### `/cnb/hooks.d/node.Dockerfile.buildpacks` -``` -com.example.node +[ -f Gemfile.lock ] && cp "$BUILDPACK_DIR/Dockerfile-ruby" "$1/Dockerfile" +[ -f package.json ] && cp "$BUILDPACK_DIR/Dockerfile-node" "$1/Dockerfile" ``` @@ -193,17 +186,6 @@ RUN curl -L https://example.com/mypkg-install | sh # installs to /opt ``` (note: rebasing is explicitly allowed because only a single directory in /opt is created) -### Future: Dynamic Run-time Base Image Selection - -The outcomes targeted by https://github.com/buildpacks/rfcs/pull/174 could be achieved with an additional type of hook. - -For example, we could introduce a `ref` format that replaces the image instead of extending it: -``` -/cnb/hooks.d/myhook.run.ref # contains: index.docker.io/example/my-ruby-run-image -/cnb/hooks.d/myhook.run.buildpacks # contains: com.example.ruby-buildpack -``` - -This would simplify buildpacks by removing the need for PURLs outside of SBoM generation. The `packages` table could be removed from both buildpack.toml and `/bin/detect`. # Drawbacks [drawbacks]: #drawbacks @@ -214,8 +196,7 @@ This would simplify buildpacks by removing the need for PURLs outside of SBoM ge [alternatives]: #alternatives - Stackpacks -- "Stackpacks-lite" -- Instead of `.buildpacks`, have hooks participate in buildpack detection (provide-only, must come first) -- Use directories for hooks: `/cnb/hooks.d/app.build.Dockerfile.out` -> `/cnb/hooks.d/app/build.Dockerfile.out` +- Previous versions of this RFC that don't interact with the buildpack API. # Unresolved Questions [unresolved-questions]: #unresolved-questions From ba9ebd7d619a60264bb117e4d69c9dbae2cfaf49 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Thu, 29 Jul 2021 18:52:35 -0400 Subject: [PATCH 11/45] RFC: Support Dockerfiles: Add build args + runtime SBoM Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 43e8fca07..9086a003b 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -78,14 +78,20 @@ However, - If a hook is missing `/bin/build`, the hook root is treated as a pre-populated `` directory. After `/bin/build` executes, the `` directory may contain -- `build.toml`, with the same contents as a buildpack's `build.toml` +- `build.toml`, with the same contents as a normal buildpack's `build.toml`, but + - With an additional `args` table array with `name` and `value` fields that are provided as build args to `build.Dockerfile` or `Dockerfile` +- `launch.toml`, with the same contents as a normal buildpack's `launch.toml`, but + - Without the `processes` table array + - Without the `slices` table array + - With an additional `args` table array with `name` and `value` fields that are provided as build args to `run.Dockerfile` or `Dockerfile` + - Either `Dockerfile` or either or both of `build.Dockerfile` and `run.Dockerfile` Support for other instruction formats, e.g., LLB JSON files, could be added in the future. `build.Dockerfile`, `run.Dockerfile`, and `Dockerfile` target the builder image, runtime base image, or both base images, respectively. -If no Dockerfiles are present, `/bin/build` may still consume build plan entries and add metadata to `build.toml`. +If no Dockerfiles are present, `/bin/build` may still consume build plan entries and add metadata to `build.toml`/`launch.toml`. Dockerfiles are applied to their corresponding base images after all hooks are executed and before any regular buildpacks are executed. Dockerfiles are applied in the order determined during buildpack detection. @@ -95,6 +101,9 @@ The `base_image` arg allows the Dockerfile to reference the original base image. The `build_id` arg allows the Dockerfile to invalidate the cache after a certain layer and must be defaulted to `0`. When the `$build_id` arg is referenced in a `RUN` instruction, all subsequent layerrs will be rebuilt on the next build (as the value will change). +Build args specified in `build.toml` are provided to `build.Dockerfile` or `Dockerfile` (when applied to the build-time base image). +Build args specified in `launch.toml` are provided to `run.Dockerfile` or `Dockerfile` (when applied to the runtime base image). + A runtime base image may indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. In the case of builder-specified Dockerfiles, `io.buildpacks.rebasable=false` is set automatically before a runtime Dockerfile is applied and must be explicitly set to `true` if desired. Rebasing an app without this label set to `true` requires passing a new `--force` flag to `pack rebase`. Finally, base images may be statically labeled with any number of `provides` that are treated as build plan entries. From 58ed531a97e1196259ba74e95a9370dcf61f70ee Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Thu, 9 Sep 2021 14:34:49 -0400 Subject: [PATCH 12/45] Fix buildpack dir env var Signed-off-by: Stephen Levine Co-authored-by: Natalie Arellano --- text/0000-dockerfiles.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 9086a003b..bb3ea7b56 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -136,8 +136,8 @@ Note: The Dockerfiles referenced must disable rebasing, and build times will be ##### `/cnb/hooks/com.example.rpmhook/bin/build` ``` #!/bin/sh -[ -f Gemfile.lock ] && cp "$BUILDPACK_DIR/Dockerfile-ruby" "$1/Dockerfile" -[ -f package.json ] && cp "$BUILDPACK_DIR/Dockerfile-node" "$1/Dockerfile" +[ -f Gemfile.lock ] && cp "$CNB_BUILDPACK_DIR/Dockerfile-ruby" "$1/Dockerfile" +[ -f package.json ] && cp "$CNB_BUILDPACK_DIR/Dockerfile-node" "$1/Dockerfile" ``` From ebf6b2a9fc1f093604741407e291deef1506d1cd Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Thu, 9 Sep 2021 14:35:33 -0400 Subject: [PATCH 13/45] Clarify detect logic Signed-off-by: Stephen Levine Co-authored-by: Natalie Arellano --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index bb3ea7b56..d70f293df 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -64,7 +64,7 @@ Unlike buildpacks, Hooks participate in the buildpack detection process, with the same interface for `/bin/detect`. However, -- `/bin/detect` is optional for hooks, and they are assumed to pass detection when it is not presetn. +- `/bin/detect` is optional for hooks, and they are assumed to pass detection when it is not present. Just like with buildpacks, a /bin/detect that exits with a 0 exit code passes detection, and fails otherwise. - Hooks may only output `provides` entries to the build plan. They must not output `requires`. - Hooks must all proceed regular buildpacks in `order` definitions (e.g., in `builder.toml`). - Hooks are always `optional`. From 22f0ef4ad7135fd49831eef2d2ab1221c5b56dad Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Thu, 9 Sep 2021 14:41:28 -0400 Subject: [PATCH 14/45] Fix typo Signed-off-by: Stephen Levine Co-authored-by: Javier Romero --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index d70f293df..643675ea2 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -99,7 +99,7 @@ Dockerfiles are applied in the order determined during buildpack detection. All Dockerfiles are provided with `base_image` and `build_id` args. The `base_image` arg allows the Dockerfile to reference the original base image. The `build_id` arg allows the Dockerfile to invalidate the cache after a certain layer and must be defaulted to `0`. -When the `$build_id` arg is referenced in a `RUN` instruction, all subsequent layerrs will be rebuilt on the next build (as the value will change). +When the `$build_id` arg is referenced in a `RUN` instruction, all subsequent layers will be rebuilt on the next build (as the value will change). Build args specified in `build.toml` are provided to `build.Dockerfile` or `Dockerfile` (when applied to the build-time base image). Build args specified in `launch.toml` are provided to `run.Dockerfile` or `Dockerfile` (when applied to the runtime base image). From 084822e9611d46349d008c9cd64476016f614b25 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Fri, 10 Sep 2021 13:18:03 -0400 Subject: [PATCH 15/45] Use UUID for build_id Signed-off-by: Stephen Levine Co-authored-by: Natalie Arellano --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 643675ea2..bbbf5a19d 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -98,7 +98,7 @@ Dockerfiles are applied in the order determined during buildpack detection. All Dockerfiles are provided with `base_image` and `build_id` args. The `base_image` arg allows the Dockerfile to reference the original base image. -The `build_id` arg allows the Dockerfile to invalidate the cache after a certain layer and must be defaulted to `0`. +The `build_id` arg allows the Dockerfile to invalidate the cache after a certain layer and must be defaulted to `0`. The executor of the Dockerfile will provide the `build_id` as a UUID (this eliminates the need to track this variable). When the `$build_id` arg is referenced in a `RUN` instruction, all subsequent layers will be rebuilt on the next build (as the value will change). Build args specified in `build.toml` are provided to `build.Dockerfile` or `Dockerfile` (when applied to the build-time base image). From b3b38ea37a3e13fffb7305d9a5b6ad9c6013a18f Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Fri, 10 Sep 2021 13:19:08 -0400 Subject: [PATCH 16/45] Fix wording re: rebasable label Signed-off-by: Stephen Levine Co-authored-by: Natalie Arellano --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index bbbf5a19d..442973880 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -104,7 +104,7 @@ When the `$build_id` arg is referenced in a `RUN` instruction, all subsequent la Build args specified in `build.toml` are provided to `build.Dockerfile` or `Dockerfile` (when applied to the build-time base image). Build args specified in `launch.toml` are provided to `run.Dockerfile` or `Dockerfile` (when applied to the runtime base image). -A runtime base image may indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. In the case of builder-specified Dockerfiles, `io.buildpacks.rebasable=false` is set automatically before a runtime Dockerfile is applied and must be explicitly set to `true` if desired. Rebasing an app without this label set to `true` requires passing a new `--force` flag to `pack rebase`. +A runtime base image may indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. In the case of builder-specified Dockerfiles, `io.buildpacks.rebasable=false` is set automatically on the base image before a runtime Dockerfile is applied and must be explicitly set to `true` if desired. If multiple Dockerfiles are applied, all must set `io.buildpacks.rebasable=true` for the final value to be `true`. Rebasing an app without this label set to `true` requires passing a new `--force` flag to `pack rebase`. Finally, base images may be statically labeled with any number of `provides` that are treated as build plan entries. These `provides` may contain fields other than `name`, which, when mismatched with `requires`, mark the entry as `unmet` by the stack. From 6eed4b299c6a6858e81a7e2d04e7a34125a8e059 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Tue, 28 Sep 2021 22:58:41 -0400 Subject: [PATCH 17/45] Clarify phases Signed-off-by: Stephen Levine Co-authored-by: Natalie Arellano --- text/0000-dockerfiles.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 442973880..654628b5e 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -30,7 +30,15 @@ For a given application, a build that uses hooks could be optimized by creating # How it Works [how-it-works]: #how-it-works -Note: kaniko, buildah, BuildKit, or the original Docker daemon may be used to apply Dockerfiles at the platform's discretion. +Note: kaniko, buildah, BuildKit, or the original Docker daemon may be used to apply Dockerfiles at the platform's discretion. The order of operations would be something like the following: +* analyze +* detect +* restore +* run hooks' bin/build, output Dockerfiles are written to a volume +* apply Dockerfiles to run image (could run in parallel with below two) +* apply Dockerfiles to build image +* build +* export ### Dynamically-applied Dockerfiles From fde7b84d0db8864292a5868a8bce39317df2f9a3 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Tue, 28 Sep 2021 23:05:11 -0400 Subject: [PATCH 18/45] RFC: Support Dockerfiles: clarify UID/GID of executables Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 654628b5e..8c71b1fe6 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -70,7 +70,7 @@ Unlike buildpacks, - Hooks must not be included in a meta-buildpacks - Hooks must not have `order`/`group` definitions in `hook.toml` -Hooks participate in the buildpack detection process, with the same interface for `/bin/detect`. +Hooks participate in the buildpack detection process, with the same UID, GID, and interface for `/bin/detect`. However, - `/bin/detect` is optional for hooks, and they are assumed to pass detection when it is not present. Just like with buildpacks, a /bin/detect that exits with a 0 exit code passes detection, and fails otherwise. - Hooks may only output `provides` entries to the build plan. They must not output `requires`. @@ -78,7 +78,7 @@ However, - Hooks are always `optional`. Hooks generate Dockerfiles before the regular buildpack build phase. -To generate these Dockerfiles, the lifecycle executes the hook's `/bin/build` executable with the same interface as regular buildpacks. +To generate these Dockerfiles, the lifecycle executes the hook's `/bin/build` executable with the same UID, GID, and interface as regular buildpacks. However, - Hooks `/bin/build` must not write to the app directory. - Hooks `/bin/build` may be executed in parallel. From f643f8b9855f6d78dac96d1e663ef19b03ba1af7 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Tue, 28 Sep 2021 23:10:56 -0400 Subject: [PATCH 19/45] RFC: Support Dockerfiles: rename hook to extension Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 80 ++++++++++++++++++++-------------------- 1 file changed, 40 insertions(+), 40 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 8c71b1fe6..e270ad1e2 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -22,10 +22,10 @@ This RFC introduces functionality for customizing base images, as an alternative # What it is [what-it-is]: #what-it-is -This RFC proposes that we replace stackpacks with dynamically-generated build-time and runtime Dockerfiles that act as pre-build hooks. -These hooks would participate in detection and execute before the buildpack build process. +This RFC proposes that we replace stackpacks with dynamically-generated build-time and runtime Dockerfiles that act as pre-build base image extensions. +These extensions would participate in detection and execute before the buildpack build process. -For a given application, a build that uses hooks could be optimized by creating a more narrowly-scoped builder that does not contain hooks. +For a given application, a build that uses extensions could be optimized by creating a more narrowly-scoped builder that does not contain extensions. # How it Works [how-it-works]: #how-it-works @@ -34,7 +34,7 @@ Note: kaniko, buildah, BuildKit, or the original Docker daemon may be used to ap * analyze * detect * restore -* run hooks' bin/build, output Dockerfiles are written to a volume +* run extensions' bin/build, output Dockerfiles are written to a volume * apply Dockerfiles to run image (could run in parallel with below two) * apply Dockerfiles to build image * build @@ -42,48 +42,48 @@ Note: kaniko, buildah, BuildKit, or the original Docker daemon may be used to ap ### Dynamically-applied Dockerfiles -A builder image may include any number of "hook" directories in `/cnb/hooks/`. +A builder image may include any number of "extensions" directories in `/cnb/ext/`. -Hooks are similar to buildpacks: they have a `/bin/build` and `/bin/detect` executable. -However, instead of a `buildpack.toml` file, hooks have a `hook.toml` file: +Extensions are similar to buildpacks: they have a `/bin/build` and `/bin/detect` executable. +However, instead of a `buildpack.toml` file, extensions have a `extension.toml` file: ```toml api = "" -[hook] -id = "" -name = "" -version = "" -homepage = "" -description = "" +[extension] +id = "" +name = "" +version = "" +homepage = "" +description = "" keywords = [ "" ] -[[hook.licenses]] +[[extension.licenses]] type = "" uri = "" ``` -Hooks may be packaged and examined similar to buildpacks, but with analogous `pack hook` subcommands. +Extensions may be packaged and examined similar to buildpacks, but with analogous `pack extension` subcommands. -Other pack CLI commands, such as `pack builder create`, will be extended to include support for hooks. +Other pack CLI commands, such as `pack builder create`, will be extended to include support for extensions. Unlike buildpacks, -- Hooks must not be included in a meta-buildpacks -- Hooks must not have `order`/`group` definitions in `hook.toml` +- Extensions must not be included in a meta-buildpacks +- Extensions must not have `order`/`group` definitions in `extension.toml` -Hooks participate in the buildpack detection process, with the same UID, GID, and interface for `/bin/detect`. +Extensions participate in the buildpack detection process, with the same UID, GID, and interface for `/bin/detect`. However, -- `/bin/detect` is optional for hooks, and they are assumed to pass detection when it is not present. Just like with buildpacks, a /bin/detect that exits with a 0 exit code passes detection, and fails otherwise. -- Hooks may only output `provides` entries to the build plan. They must not output `requires`. -- Hooks must all proceed regular buildpacks in `order` definitions (e.g., in `builder.toml`). -- Hooks are always `optional`. +- `/bin/detect` is optional for extensions, and they are assumed to pass detection when it is not present. Just like with buildpacks, a /bin/detect that exits with a 0 exit code passes detection, and fails otherwise. +- Extensions may only output `provides` entries to the build plan. They must not output `requires`. +- Extensions must all proceed regular buildpacks in `order` definitions (e.g., in `builder.toml`). +- Extensions are always `optional`. -Hooks generate Dockerfiles before the regular buildpack build phase. -To generate these Dockerfiles, the lifecycle executes the hook's `/bin/build` executable with the same UID, GID, and interface as regular buildpacks. +Extensions generate Dockerfiles before the regular buildpack build phase. +To generate these Dockerfiles, the lifecycle executes the extension's `/bin/build` executable with the same UID, GID, and interface as regular buildpacks. However, -- Hooks `/bin/build` must not write to the app directory. -- Hooks `/bin/build` may be executed in parallel. -- Hooks `` directory is replaced by an `` directory. -- If a hook is missing `/bin/build`, the hook root is treated as a pre-populated `` directory. +- Extensions `/bin/build` must not write to the app directory. +- Extensions `/bin/build` may be executed in parallel. +- Extensions `` directory is replaced by an `` directory. +- If an extension is missing `/bin/build`, the extension root is treated as a pre-populated `` directory. After `/bin/build` executes, the `` directory may contain - `build.toml`, with the same contents as a normal buildpack's `build.toml`, but @@ -101,7 +101,7 @@ Support for other instruction formats, e.g., LLB JSON files, could be added in t If no Dockerfiles are present, `/bin/build` may still consume build plan entries and add metadata to `build.toml`/`launch.toml`. -Dockerfiles are applied to their corresponding base images after all hooks are executed and before any regular buildpacks are executed. +Dockerfiles are applied to their corresponding base images after all extensions are executed and before any regular buildpacks are executed. Dockerfiles are applied in the order determined during buildpack detection. All Dockerfiles are provided with `base_image` and `build_id` args. @@ -118,30 +118,30 @@ Finally, base images may be statically labeled with any number of `provides` tha These `provides` may contain fields other than `name`, which, when mismatched with `requires`, mark the entry as `unmet` by the stack. This is important to ensure that: - Rebasing always remains an option for end users. -- Buildpacks do not become dependent on hooks. +- Buildpacks do not become dependent on extensions. - Builds can be time-optimized by creating base images ahead of time. -NOTE: the above can be accomplished by a hook with a no-op `/bin/build` -- do we really need this? +NOTE: the above can be accomplished by an extension with a no-op `/bin/build` -- do we really need this? -#### Example: App-specified Dockerfile Hook +#### Example: App-specified Dockerfile Extension -This example hook would allow an app to provide runtime and build-time base image extensions as "run.Dockerfile" and "build.Dockerfile." +This example extension would allow an app to provide runtime and build-time base image extensions as "run.Dockerfile" and "build.Dockerfile." The app developer can decide whether the extensions are rebasable. -##### `/cnb/hooks/com.example.apphook/bin/build` +##### `/cnb/ext/com.example.appext/bin/build` ``` #!/bin/sh [ -f build.Dockerfile ] && cp build.Dockerfile "$1/" [ -f run.Dockerfile ] && cp run.Dockerfile "$1/" ``` -#### Example: RPM Dockerfile Hook (app-based) +#### Example: RPM Dockerfile Extension (app-based) -This example hook would allow a builder to install RPMs for each language runtime, based on the app directory. +This example extension would allow a builder to install RPMs for each language runtime, based on the app directory. Note: The Dockerfiles referenced must disable rebasing, and build times will be slower compared to buildpack-provided runtimes. -##### `/cnb/hooks/com.example.rpmhook/bin/build` +##### `/cnb/ext/com.example.rpmext/bin/build` ``` #!/bin/sh [ -f Gemfile.lock ] && cp "$CNB_BUILDPACK_DIR/Dockerfile-ruby" "$1/Dockerfile" @@ -179,7 +179,7 @@ USER ${CNB_USER_ID}:${CNB_GROUP_ID} COPY genpkgs /cnb/image/genpkgs ``` -`run.Dockerfile` for use with the example `app.run.Dockerfile.out` hook that always installs the latest version of curl: +`run.Dockerfile` for use with the example `app.run.Dockerfile.out` extension that always installs the latest version of curl: ``` ARG base_image FROM ${base_image} @@ -191,7 +191,7 @@ RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/* ``` (note: this Dockerfile disables rebasing, as OS package installation is not rebasable) -`run.Dockerfile` for use with the example `app.run.Dockerfile.out` hook that installs a special package to /opt: +`run.Dockerfile` for use with the example `app.run.Dockerfile.out` extension that installs a special package to /opt: ``` ARG base_image FROM ${base_image} From 3a37d5483c8066a55f5cbf6c7a2c2968aa3b48bb Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Tue, 28 Sep 2021 23:17:24 -0400 Subject: [PATCH 20/45] RFC: Support Dockerfiles: fix typo Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index e270ad1e2..17a48676f 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -74,7 +74,7 @@ Extensions participate in the buildpack detection process, with the same UID, GI However, - `/bin/detect` is optional for extensions, and they are assumed to pass detection when it is not present. Just like with buildpacks, a /bin/detect that exits with a 0 exit code passes detection, and fails otherwise. - Extensions may only output `provides` entries to the build plan. They must not output `requires`. -- Extensions must all proceed regular buildpacks in `order` definitions (e.g., in `builder.toml`). +- Extensions must all precede regular buildpacks in `order` definitions (e.g., in `builder.toml`). - Extensions are always `optional`. Extensions generate Dockerfiles before the regular buildpack build phase. From f513c24dd381e285b92c4aa6504bd4bcca74114c Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Tue, 28 Sep 2021 23:27:18 -0400 Subject: [PATCH 21/45] RFC: Support Dockerfiles: remove static provides Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 17a48676f..22b1eeabc 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -114,15 +114,6 @@ Build args specified in `launch.toml` are provided to `run.Dockerfile` or `Docke A runtime base image may indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. In the case of builder-specified Dockerfiles, `io.buildpacks.rebasable=false` is set automatically on the base image before a runtime Dockerfile is applied and must be explicitly set to `true` if desired. If multiple Dockerfiles are applied, all must set `io.buildpacks.rebasable=true` for the final value to be `true`. Rebasing an app without this label set to `true` requires passing a new `--force` flag to `pack rebase`. -Finally, base images may be statically labeled with any number of `provides` that are treated as build plan entries. -These `provides` may contain fields other than `name`, which, when mismatched with `requires`, mark the entry as `unmet` by the stack. -This is important to ensure that: -- Rebasing always remains an option for end users. -- Buildpacks do not become dependent on extensions. -- Builds can be time-optimized by creating base images ahead of time. - -NOTE: the above can be accomplished by an extension with a no-op `/bin/build` -- do we really need this? - #### Example: App-specified Dockerfile Extension This example extension would allow an app to provide runtime and build-time base image extensions as "run.Dockerfile" and "build.Dockerfile." @@ -219,6 +210,7 @@ RUN curl -L https://example.com/mypkg-install | sh # installs to /opt [unresolved-questions]: #unresolved-questions - Should `genpkgs` be part of this proposal? Opinion: Yes, otherwise it's difficult to maintain a valid SBoM. +- Should we allow base images to provide build plan entries so that extensions aren't required to satisfy buildpacks? Opinion: Not yet, no-op extension can be used for now. # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From 884eaef5582e5c30b8cacbcdbdd82514f70edc24 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 16 Feb 2022 14:30:35 -0500 Subject: [PATCH 22/45] Clarify Dockerfile restrictions Signed-off-by: Stephen Levine Co-authored-by: Natalie Arellano --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 22b1eeabc..2c76f9130 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -102,7 +102,7 @@ Support for other instruction formats, e.g., LLB JSON files, could be added in t If no Dockerfiles are present, `/bin/build` may still consume build plan entries and add metadata to `build.toml`/`launch.toml`. Dockerfiles are applied to their corresponding base images after all extensions are executed and before any regular buildpacks are executed. -Dockerfiles are applied in the order determined during buildpack detection. +Dockerfiles are applied in the order determined during buildpack detection. When multiple Dockerfiles are applied, the intermediate image generated from the application of the current Dockerfile will be provided as the `base_image` ARG to the next Dockerfile. Dockerfiles that target the run image (only) may ignore the provided `base_image` (e.g., `FROM some-other-image`). Dockerfiles that change the runtime base image may still use `COPY --from=${base_image}`. All Dockerfiles are provided with `base_image` and `build_id` args. The `base_image` arg allows the Dockerfile to reference the original base image. From cd9471f7c8f0c84a14d5b2d9a1111979808e4df7 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Tue, 1 Mar 2022 16:25:05 -0500 Subject: [PATCH 23/45] Move extensions out of order table Signed-off-by: Stephen Levine Co-authored-by: Natalie Arellano --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 2c76f9130..f66f7c30f 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -74,7 +74,7 @@ Extensions participate in the buildpack detection process, with the same UID, GI However, - `/bin/detect` is optional for extensions, and they are assumed to pass detection when it is not present. Just like with buildpacks, a /bin/detect that exits with a 0 exit code passes detection, and fails otherwise. - Extensions may only output `provides` entries to the build plan. They must not output `requires`. -- Extensions must all precede regular buildpacks in `order` definitions (e.g., in `builder.toml`). +- Extensions are not included in `order` definitions (e.g., in `builder.toml`); instead, a separate `order-ext` table should be used. The `order-ext` table will be prepended to the provided `order` (as if `order-ext` were a meta-buildpack). - Extensions are always `optional`. Extensions generate Dockerfiles before the regular buildpack build phase. From a2c06a553bf822fdd6d3aaf2e23139e79eaa4b96 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:32:53 -0400 Subject: [PATCH 24/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index f66f7c30f..800928f0a 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -33,10 +33,10 @@ For a given application, a build that uses extensions could be optimized by crea Note: kaniko, buildah, BuildKit, or the original Docker daemon may be used to apply Dockerfiles at the platform's discretion. The order of operations would be something like the following: * analyze * detect -* restore -* run extensions' bin/build, output Dockerfiles are written to a volume -* apply Dockerfiles to run image (could run in parallel with below two) +* run extensions' bin/generate, output Dockerfiles are written to a volume +* apply Dockerfiles to run image (could run in parallel with build image extension) * apply Dockerfiles to build image +* restore * build * export From 2935c8a3dd2dccbc1a28bd370e5b6ff3df74a9fc Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:33:03 -0400 Subject: [PATCH 25/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 800928f0a..0e6b8174a 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -44,7 +44,7 @@ Note: kaniko, buildah, BuildKit, or the original Docker daemon may be used to ap A builder image may include any number of "extensions" directories in `/cnb/ext/`. -Extensions are similar to buildpacks: they have a `/bin/build` and `/bin/detect` executable. +Extensions are similar to buildpacks: they have two executables: `/bin/detect` and `/bin/generate`. The interface for these executables is similar to a buildpack's `/bin/detect` and `/bin/build`. However, instead of a `buildpack.toml` file, extensions have a `extension.toml` file: ```toml api = "" From 01786f8ee6846acbfc73382e82809ffd770c4940 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:34:11 -0400 Subject: [PATCH 26/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 0e6b8174a..1000da8e8 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -73,6 +73,7 @@ Unlike buildpacks, Extensions participate in the buildpack detection process, with the same UID, GID, and interface for `/bin/detect`. However, - `/bin/detect` is optional for extensions, and they are assumed to pass detection when it is not present. Just like with buildpacks, a /bin/detect that exits with a 0 exit code passes detection, and fails otherwise. +- If an extension is missing `/bin/detect`, the extension root is treated as a pre-populated output directory (i.e., extensions can include a static build plan). - Extensions may only output `provides` entries to the build plan. They must not output `requires`. - Extensions are not included in `order` definitions (e.g., in `builder.toml`); instead, a separate `order-ext` table should be used. The `order-ext` table will be prepended to the provided `order` (as if `order-ext` were a meta-buildpack). - Extensions are always `optional`. From 10a02f28f1907f85e536951e86a089afe987e1c1 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:34:24 -0400 Subject: [PATCH 27/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 1000da8e8..5aa41f09a 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -75,7 +75,7 @@ However, - `/bin/detect` is optional for extensions, and they are assumed to pass detection when it is not present. Just like with buildpacks, a /bin/detect that exits with a 0 exit code passes detection, and fails otherwise. - If an extension is missing `/bin/detect`, the extension root is treated as a pre-populated output directory (i.e., extensions can include a static build plan). - Extensions may only output `provides` entries to the build plan. They must not output `requires`. -- Extensions are not included in `order` definitions (e.g., in `builder.toml`); instead, a separate `order-ext` table should be used. The `order-ext` table will be prepended to the provided `order` (as if `order-ext` were a meta-buildpack). +- Extensions are not included in `order` definitions (e.g., in `builder.toml`); instead, a separate `order-extensions` table should be used. The `order-extensions` table will be prepended to each group in the provided `order` (as if `order-extensions` were a composite buildpack). - Extensions are always `optional`. Extensions generate Dockerfiles before the regular buildpack build phase. From 2c7bb58fbfc4177e97ee8a33759a4f055674c988 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:34:30 -0400 Subject: [PATCH 28/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 5aa41f09a..6e0c8e176 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -81,7 +81,7 @@ However, Extensions generate Dockerfiles before the regular buildpack build phase. To generate these Dockerfiles, the lifecycle executes the extension's `/bin/build` executable with the same UID, GID, and interface as regular buildpacks. However, -- Extensions `/bin/build` must not write to the app directory. +- Extensions `/bin/generate` must not write to the app directory. - Extensions `/bin/build` may be executed in parallel. - Extensions `` directory is replaced by an `` directory. - If an extension is missing `/bin/build`, the extension root is treated as a pre-populated `` directory. From 2c6ca724788cf861ad5dd142a7b8e24efa3bb341 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:35:25 -0400 Subject: [PATCH 29/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 1 - 1 file changed, 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 6e0c8e176..4588ea493 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -82,7 +82,6 @@ Extensions generate Dockerfiles before the regular buildpack build phase. To generate these Dockerfiles, the lifecycle executes the extension's `/bin/build` executable with the same UID, GID, and interface as regular buildpacks. However, - Extensions `/bin/generate` must not write to the app directory. -- Extensions `/bin/build` may be executed in parallel. - Extensions `` directory is replaced by an `` directory. - If an extension is missing `/bin/build`, the extension root is treated as a pre-populated `` directory. From aa74ace4c16fd53174bf12547f4733a4b3b46d03 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:35:34 -0400 Subject: [PATCH 30/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 4588ea493..9c13d95a7 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -99,7 +99,7 @@ Support for other instruction formats, e.g., LLB JSON files, could be added in t `build.Dockerfile`, `run.Dockerfile`, and `Dockerfile` target the builder image, runtime base image, or both base images, respectively. -If no Dockerfiles are present, `/bin/build` may still consume build plan entries and add metadata to `build.toml`/`launch.toml`. +If no Dockerfiles are present, `/bin/generate` may still consume build plan entries and add metadata to `build.toml`/`launch.toml`. Dockerfiles are applied to their corresponding base images after all extensions are executed and before any regular buildpacks are executed. Dockerfiles are applied in the order determined during buildpack detection. When multiple Dockerfiles are applied, the intermediate image generated from the application of the current Dockerfile will be provided as the `base_image` ARG to the next Dockerfile. Dockerfiles that target the run image (only) may ignore the provided `base_image` (e.g., `FROM some-other-image`). Dockerfiles that change the runtime base image may still use `COPY --from=${base_image}`. From 10b6c9fce0b12c2215f6ae5b8726afe846b5d72c Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:35:41 -0400 Subject: [PATCH 31/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 9c13d95a7..f0d15d83c 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -85,7 +85,7 @@ However, - Extensions `` directory is replaced by an `` directory. - If an extension is missing `/bin/build`, the extension root is treated as a pre-populated `` directory. -After `/bin/build` executes, the `` directory may contain +After `/bin/generate` executes, the `` directory may contain - `build.toml`, with the same contents as a normal buildpack's `build.toml`, but - With an additional `args` table array with `name` and `value` fields that are provided as build args to `build.Dockerfile` or `Dockerfile` - `launch.toml`, with the same contents as a normal buildpack's `launch.toml`, but From 1cc27898aecba63d16bc0453a12068364281122e Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:37:20 -0400 Subject: [PATCH 32/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index f0d15d83c..c66584913 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -119,7 +119,7 @@ A runtime base image may indicate that it preserves ABI compatibility by adding This example extension would allow an app to provide runtime and build-time base image extensions as "run.Dockerfile" and "build.Dockerfile." The app developer can decide whether the extensions are rebasable. -##### `/cnb/ext/com.example.appext/bin/build` +##### `/cnb/ext/com.example.appext/bin/generate` ``` #!/bin/sh [ -f build.Dockerfile ] && cp build.Dockerfile "$1/" From a2282f17aab2cb817776a86e5ca1559269d4ea80 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:37:28 -0400 Subject: [PATCH 33/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index c66584913..1e5f11b95 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -132,7 +132,7 @@ This example extension would allow a builder to install RPMs for each language r Note: The Dockerfiles referenced must disable rebasing, and build times will be slower compared to buildpack-provided runtimes. -##### `/cnb/ext/com.example.rpmext/bin/build` +##### `/cnb/ext/com.example.rpmext/bin/generate` ``` #!/bin/sh [ -f Gemfile.lock ] && cp "$CNB_BUILDPACK_DIR/Dockerfile-ruby" "$1/Dockerfile" From 9ba0bc23a42a34f9e68d87dc4be40d0d95e4fbe9 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:37:39 -0400 Subject: [PATCH 34/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 1e5f11b95..166da5510 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -144,7 +144,7 @@ Note: The Dockerfiles referenced must disable rebasing, and build times will be The same Dockerfile format may be used to create new base images or modify existing base images outside of the app build process (e.g., before creating a builder). Any specified labels override existing values. -Dockerfiles that are used to create a base image must create a `/cnb/image/genpkgs` executable that outputs a [CycloneDX](https://cyclonedx.org)-formatted list of packages in the image with PURL IDs when invoked. This executable is executed after all Dockerfiles are applied, and the output replaces the label `io.buildpacks.sbom`. This label doubles as a Software Bill-of-Materials for the base image. In the future, this label will serve as a starting point for the application SBoM. +The project will provide tooling that can be used to scan the extended run image. For more information, see https://github.com/buildpacks/rfcs/pull/195. ### Example Dockerfiles From 95336429e52113fb609c753dfe06b72a3416949e Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:37:52 -0400 Subject: [PATCH 35/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 166da5510..1e2223230 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -42,7 +42,7 @@ Note: kaniko, buildah, BuildKit, or the original Docker daemon may be used to ap ### Dynamically-applied Dockerfiles -A builder image may include any number of "extensions" directories in `/cnb/ext/`. +A builder image may include any number of "extensions" directories in `/cnb/extensions/`. Extensions are similar to buildpacks: they have two executables: `/bin/detect` and `/bin/generate`. The interface for these executables is similar to a buildpack's `/bin/detect` and `/bin/build`. However, instead of a `buildpack.toml` file, extensions have a `extension.toml` file: From bfc5781f99ad2ac4d3a301a4185db806f57b48cd Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:38:00 -0400 Subject: [PATCH 36/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 1e2223230..1825510c9 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -79,7 +79,7 @@ However, - Extensions are always `optional`. Extensions generate Dockerfiles before the regular buildpack build phase. -To generate these Dockerfiles, the lifecycle executes the extension's `/bin/build` executable with the same UID, GID, and interface as regular buildpacks. +To generate these Dockerfiles, the lifecycle executes the extension's `/bin/generate` executable with the same UID, GID, and interface as regular buildpacks. However, - Extensions `/bin/generate` must not write to the app directory. - Extensions `` directory is replaced by an `` directory. From 68773ac588014426e0a51f07230c94eb1b3eaced Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:38:32 -0400 Subject: [PATCH 37/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 1825510c9..11cf5d3dc 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -83,7 +83,7 @@ To generate these Dockerfiles, the lifecycle executes the extension's `/bin/gene However, - Extensions `/bin/generate` must not write to the app directory. - Extensions `` directory is replaced by an `` directory. -- If an extension is missing `/bin/build`, the extension root is treated as a pre-populated `` directory. +- If an extension is missing `/bin/generate`, the extension root is treated as a pre-populated `` directory. After `/bin/generate` executes, the `` directory may contain - `build.toml`, with the same contents as a normal buildpack's `build.toml`, but From fefd3388727c3f36a63a02640e1335ea059e8729 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:39:02 -0400 Subject: [PATCH 38/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 11cf5d3dc..73f253f16 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -166,8 +166,6 @@ RUN groupadd cnb --gid ${CNB_GROUP_ID} && \ useradd --uid ${CNB_USER_ID} --gid ${CNB_GROUP_ID} -m -s /bin/bash cnb USER ${CNB_USER_ID}:${CNB_GROUP_ID} - -COPY genpkgs /cnb/image/genpkgs ``` `run.Dockerfile` for use with the example `app.run.Dockerfile.out` extension that always installs the latest version of curl: From 74767a099dc754369c34a8f744f2583a8ae8905d Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:39:10 -0400 Subject: [PATCH 39/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 1 - 1 file changed, 1 deletion(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 73f253f16..bca24fec0 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -207,7 +207,6 @@ RUN curl -L https://example.com/mypkg-install | sh # installs to /opt # Unresolved Questions [unresolved-questions]: #unresolved-questions -- Should `genpkgs` be part of this proposal? Opinion: Yes, otherwise it's difficult to maintain a valid SBoM. - Should we allow base images to provide build plan entries so that extensions aren't required to satisfy buildpacks? Opinion: Not yet, no-op extension can be used for now. # Spec. Changes (OPTIONAL) From 3b73177c87dc6d489f9247612fe03bd3234f1103 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 22 Jun 2022 13:39:18 -0400 Subject: [PATCH 40/45] Update text/0000-dockerfiles.md Co-authored-by: Natalie Arellano Signed-off-by: Stephen Levine --- text/0000-dockerfiles.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index bca24fec0..89a7cdf30 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -213,3 +213,14 @@ RUN curl -L https://example.com/mypkg-install | sh # installs to /opt [spec-changes]: #spec-changes This RFC requires extensive changes to all specifications. + +To deliver incremental value and gather feedback as we implement this large feature, a phased implementation is proposed: +* Phase 1: run.Dockerfiles can be used to switch the runtime base image +* Phase 2: build.Dockerfiles can be used to extend the build time base image + * 2a: `pack` applies the build.Dockerfiles + * 2b: the lifecycle applies the build.Dockerfiles using kaniko +* Phase 3: Dockerfiles and / or run.Dockerfiles can be used to extend the runtime base image + * 3a: `pack` applies the run.Dockerfiles + * 3b: the lifecycle applies the run.Dockerfiles using kaniko + +https://github.com/buildpacks/spec/pull/307 and https://github.com/buildpacks/spec/pull/308 describe the spec changes needed for phase 1. https://github.com/buildpacks/spec/pull/298 approximately describes the spec changes needed for all phases. From 4abb253d219eb6cc1dd32d06e4af3f3a3f801760 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Tue, 5 Jul 2022 15:46:24 -0400 Subject: [PATCH 41/45] Add a note about resolving registry credentials up front Signed-off-by: Natalie Arellano --- text/0000-dockerfiles.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 89a7cdf30..4539e1b81 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -40,6 +40,8 @@ Note: kaniko, buildah, BuildKit, or the original Docker daemon may be used to ap * build * export +When Dockerfiles are used to update the run image, care should be taken to ensure registry access prior to the `build` phase, to avoid long builds that fail due to incorrect credentials. + ### Dynamically-applied Dockerfiles A builder image may include any number of "extensions" directories in `/cnb/extensions/`. From f8e7bfa555c302ac9203e8ccfa0ab29e7987e5c3 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Wed, 6 Jul 2022 11:24:44 -0400 Subject: [PATCH 42/45] Updates per Emily's feedback Signed-off-by: Natalie Arellano --- text/0000-dockerfiles.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 4539e1b81..6f5afb977 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -32,10 +32,9 @@ For a given application, a build that uses extensions could be optimized by crea Note: kaniko, buildah, BuildKit, or the original Docker daemon may be used to apply Dockerfiles at the platform's discretion. The order of operations would be something like the following: * analyze -* detect -* run extensions' bin/generate, output Dockerfiles are written to a volume -* apply Dockerfiles to run image (could run in parallel with build image extension) -* apply Dockerfiles to build image +* detect - after standard detection detect will also run extensions' bin/generate, output Dockerfiles are written to a volume +* apply Dockerfiles to run image (could run in parallel with builder image extension) +* apply Dockerfiles to builder image * restore * build * export @@ -88,9 +87,10 @@ However, - If an extension is missing `/bin/generate`, the extension root is treated as a pre-populated `` directory. After `/bin/generate` executes, the `` directory may contain -- `build.toml`, with the same contents as a normal buildpack's `build.toml`, but +- `build.toml`, with the same contents as a normal buildpack's `build.toml` (the `unmet` table array), but - With an additional `args` table array with `name` and `value` fields that are provided as build args to `build.Dockerfile` or `Dockerfile` -- `launch.toml`, with the same contents as a normal buildpack's `launch.toml`, but +- `launch.toml`, + - Without the `labels` table array - Without the `processes` table array - Without the `slices` table array - With an additional `args` table array with `name` and `value` fields that are provided as build args to `run.Dockerfile` or `Dockerfile` @@ -101,7 +101,7 @@ Support for other instruction formats, e.g., LLB JSON files, could be added in t `build.Dockerfile`, `run.Dockerfile`, and `Dockerfile` target the builder image, runtime base image, or both base images, respectively. -If no Dockerfiles are present, `/bin/generate` may still consume build plan entries and add metadata to `build.toml`/`launch.toml`. +If no Dockerfiles are present, `/bin/generate` may still consume build plan entries. Dockerfiles are applied to their corresponding base images after all extensions are executed and before any regular buildpacks are executed. Dockerfiles are applied in the order determined during buildpack detection. When multiple Dockerfiles are applied, the intermediate image generated from the application of the current Dockerfile will be provided as the `base_image` ARG to the next Dockerfile. Dockerfiles that target the run image (only) may ignore the provided `base_image` (e.g., `FROM some-other-image`). Dockerfiles that change the runtime base image may still use `COPY --from=${base_image}`. From 3a26570c2c4a2634b14832282828178a38782493 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Thu, 7 Jul 2022 11:23:03 -0400 Subject: [PATCH 43/45] Updates per 7/7/22 Working Group - Change launch.toml -> run.toml since there are no overlapping fields between buildpacks and extensions - Remove Dockerfile without a prefix since the args must be specified in either build.toml or run.toml - Change must -> should for defaulting build_id - Nest pre-populated output files under detect or generate (future proofing) - Mention how we'll preserve top layer information for rebase Signed-off-by: Natalie Arellano --- text/0000-dockerfiles.md | 52 +++++++++++++++++++--------------------- 1 file changed, 24 insertions(+), 28 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 6f5afb977..e6cdcbc28 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -17,7 +17,7 @@ This RFC introduces functionality for customizing base images, as an alternative # Motivation [motivation]: #motivation -[RFC0069](https://github.com/buildpacks/rfcs/blob/main/text/0069-stack-buildpacks.md) introduces complexity by defining an API that allows buildpacks to modify base images. To avoid this complexity, we could rely on generated Dockerfiles for base image manipulation. This would simplify the original proposal by, e.g., only requiring a copy of the buildpack on the build-time base image. +[RFC0069](https://github.com/buildpacks/rfcs/blob/main/text/0069-stack-buildpacks.md) introduces complexity by defining an API that allows buildpacks to modify base images. To avoid this complexity, we could rely on generated Dockerfiles for base image manipulation. This would simplify the original proposal by, e.g., only requiring a copy of the extension on the build-time base image. # What it is [what-it-is]: #what-it-is @@ -31,13 +31,13 @@ For a given application, a build that uses extensions could be optimized by crea [how-it-works]: #how-it-works Note: kaniko, buildah, BuildKit, or the original Docker daemon may be used to apply Dockerfiles at the platform's discretion. The order of operations would be something like the following: -* analyze -* detect - after standard detection detect will also run extensions' bin/generate, output Dockerfiles are written to a volume -* apply Dockerfiles to run image (could run in parallel with builder image extension) -* apply Dockerfiles to builder image -* restore -* build -* export +* `analyze` +* `detect` - after standard detection detect will also run extensions' bin/generate, output Dockerfiles are written to a volume +* `extend` - applies run.Dockerfiles to run image (could run in parallel with builder image extension) +* `extend` - applies build.Dockerfiles to builder image +* `restore` +* `build` +* `export` When Dockerfiles are used to update the run image, care should be taken to ensure registry access prior to the `build` phase, to avoid long builds that fail due to incorrect credentials. @@ -74,7 +74,7 @@ Unlike buildpacks, Extensions participate in the buildpack detection process, with the same UID, GID, and interface for `/bin/detect`. However, - `/bin/detect` is optional for extensions, and they are assumed to pass detection when it is not present. Just like with buildpacks, a /bin/detect that exits with a 0 exit code passes detection, and fails otherwise. -- If an extension is missing `/bin/detect`, the extension root is treated as a pre-populated output directory (i.e., extensions can include a static build plan). +- If an extension is missing `/bin/detect`, the extension root `./detect` directory is treated as a pre-populated output directory (i.e., extensions can include a static build plan). - Extensions may only output `provides` entries to the build plan. They must not output `requires`. - Extensions are not included in `order` definitions (e.g., in `builder.toml`); instead, a separate `order-extensions` table should be used. The `order-extensions` table will be prepended to each group in the provided `order` (as if `order-extensions` were a composite buildpack). - Extensions are always `optional`. @@ -84,22 +84,18 @@ To generate these Dockerfiles, the lifecycle executes the extension's `/bin/gene However, - Extensions `/bin/generate` must not write to the app directory. - Extensions `` directory is replaced by an `` directory. -- If an extension is missing `/bin/generate`, the extension root is treated as a pre-populated `` directory. +- If an extension is missing `/bin/generate`, the extension root `./generate` directory is treated as a pre-populated `` directory. After `/bin/generate` executes, the `` directory may contain - `build.toml`, with the same contents as a normal buildpack's `build.toml` (the `unmet` table array), but - - With an additional `args` table array with `name` and `value` fields that are provided as build args to `build.Dockerfile` or `Dockerfile` -- `launch.toml`, - - Without the `labels` table array - - Without the `processes` table array - - Without the `slices` table array - - With an additional `args` table array with `name` and `value` fields that are provided as build args to `run.Dockerfile` or `Dockerfile` - -- Either `Dockerfile` or either or both of `build.Dockerfile` and `run.Dockerfile` + - With an additional `args` table array with `name` and `value` fields that are provided as build args to `build.Dockerfile` +- `run.toml`, + - With an `args` table array with `name` and `value` fields that are provided as build args to `run.Dockerfile` +- Either or both of `build.Dockerfile` and `run.Dockerfile` Support for other instruction formats, e.g., LLB JSON files, could be added in the future. -`build.Dockerfile`, `run.Dockerfile`, and `Dockerfile` target the builder image, runtime base image, or both base images, respectively. +`build.Dockerfile` and `run.Dockerfile`target the builder image or runtime base image, respectively. If no Dockerfiles are present, `/bin/generate` may still consume build plan entries. @@ -108,13 +104,13 @@ Dockerfiles are applied in the order determined during buildpack detection. When All Dockerfiles are provided with `base_image` and `build_id` args. The `base_image` arg allows the Dockerfile to reference the original base image. -The `build_id` arg allows the Dockerfile to invalidate the cache after a certain layer and must be defaulted to `0`. The executor of the Dockerfile will provide the `build_id` as a UUID (this eliminates the need to track this variable). +The `build_id` arg allows the Dockerfile to invalidate the cache after a certain layer and should be defaulted to `0`. The executor of the Dockerfile will provide the `build_id` as a UUID (this eliminates the need to track this variable). When the `$build_id` arg is referenced in a `RUN` instruction, all subsequent layers will be rebuilt on the next build (as the value will change). -Build args specified in `build.toml` are provided to `build.Dockerfile` or `Dockerfile` (when applied to the build-time base image). -Build args specified in `launch.toml` are provided to `run.Dockerfile` or `Dockerfile` (when applied to the runtime base image). +Build args specified in `build.toml` are provided to `build.Dockerfile` (when applied to the build-time base image). +Build args specified in `run.toml` are provided to `run.Dockerfile` (when applied to the runtime base image). -A runtime base image may indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. In the case of builder-specified Dockerfiles, `io.buildpacks.rebasable=false` is set automatically on the base image before a runtime Dockerfile is applied and must be explicitly set to `true` if desired. If multiple Dockerfiles are applied, all must set `io.buildpacks.rebasable=true` for the final value to be `true`. Rebasing an app without this label set to `true` requires passing a new `--force` flag to `pack rebase`. +A runtime base image may indicate that it preserves ABI compatibility by adding the label `io.buildpacks.rebasable=true`. In the case of builder-specified Dockerfiles, `io.buildpacks.rebasable=false` is set automatically on the base image before a runtime Dockerfile is applied and must be explicitly set to `true` if desired. If multiple Dockerfiles are applied, all must set `io.buildpacks.rebasable=true` for the final value to be `true`. Rebasing an app without this label set to `true` requires passing a new `--force` flag to `pack rebase`. When the run image is extended and `io.buildpacks.rebasable=true`, the `extend` phase will communicate to the `export` phase the top layer of the run image (prior to extension) so that the exporter can set the appropriate value of `io.buildpacks.lifecycle.metadata` `runImage.topLayer`. #### Example: App-specified Dockerfile Extension @@ -137,8 +133,8 @@ Note: The Dockerfiles referenced must disable rebasing, and build times will be ##### `/cnb/ext/com.example.rpmext/bin/generate` ``` #!/bin/sh -[ -f Gemfile.lock ] && cp "$CNB_BUILDPACK_DIR/Dockerfile-ruby" "$1/Dockerfile" -[ -f package.json ] && cp "$CNB_BUILDPACK_DIR/Dockerfile-node" "$1/Dockerfile" +[ -f Gemfile.lock ] && cp "$CNB_BUILDPACK_DIR/Dockerfile-ruby" "$1/build.Dockerfile" +[ -f package.json ] && cp "$CNB_BUILDPACK_DIR/Dockerfile-node" "$1/build.Dockerfile" ``` @@ -170,7 +166,7 @@ RUN groupadd cnb --gid ${CNB_GROUP_ID} && \ USER ${CNB_USER_ID}:${CNB_GROUP_ID} ``` -`run.Dockerfile` for use with the example `app.run.Dockerfile.out` extension that always installs the latest version of curl: +`run.Dockerfile` that always installs the latest version of curl: ``` ARG base_image FROM ${base_image} @@ -182,7 +178,7 @@ RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/* ``` (note: this Dockerfile disables rebasing, as OS package installation is not rebasable) -`run.Dockerfile` for use with the example `app.run.Dockerfile.out` extension that installs a special package to /opt: +`run.Dockerfile` that installs a special package to /opt: ``` ARG base_image FROM ${base_image} @@ -221,7 +217,7 @@ To deliver incremental value and gather feedback as we implement this large feat * Phase 2: build.Dockerfiles can be used to extend the build time base image * 2a: `pack` applies the build.Dockerfiles * 2b: the lifecycle applies the build.Dockerfiles using kaniko -* Phase 3: Dockerfiles and / or run.Dockerfiles can be used to extend the runtime base image +* Phase 3: run.Dockerfiles can be used to extend the runtime base image * 3a: `pack` applies the run.Dockerfiles * 3b: the lifecycle applies the run.Dockerfiles using kaniko From 73de1832092fff1dff0d827f9323744cb3140223 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Wed, 13 Jul 2022 15:37:20 -0400 Subject: [PATCH 44/45] Apply suggestions from code review Co-authored-by: Emily Casey Signed-off-by: Natalie Arellano --- text/0000-dockerfiles.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index e6cdcbc28..23e9eebd7 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -45,7 +45,7 @@ When Dockerfiles are used to update the run image, care should be taken to ensur A builder image may include any number of "extensions" directories in `/cnb/extensions/`. -Extensions are similar to buildpacks: they have two executables: `/bin/detect` and `/bin/generate`. The interface for these executables is similar to a buildpack's `/bin/detect` and `/bin/build`. +Extensions are similar to buildpacks: they have two executables: `./bin/detect` and `./bin/generate`. The interface for these executables is similar to a buildpack's `./bin/detect` and `./bin/build`. However, instead of a `buildpack.toml` file, extensions have a `extension.toml` file: ```toml api = "" @@ -71,22 +71,22 @@ Unlike buildpacks, - Extensions must not be included in a meta-buildpacks - Extensions must not have `order`/`group` definitions in `extension.toml` -Extensions participate in the buildpack detection process, with the same UID, GID, and interface for `/bin/detect`. +Extensions participate in the buildpack detection process, with the same UID, GID, and interface for `./bin/detect`. However, -- `/bin/detect` is optional for extensions, and they are assumed to pass detection when it is not present. Just like with buildpacks, a /bin/detect that exits with a 0 exit code passes detection, and fails otherwise. -- If an extension is missing `/bin/detect`, the extension root `./detect` directory is treated as a pre-populated output directory (i.e., extensions can include a static build plan). +- `./bin/detect` is optional for extensions, and they are assumed to pass detection when it is not present. Just like with buildpacks, a `./bin/detect` that exits with a 0 exit code passes detection, and fails otherwise. +- If an extension is missing `./bin/detect`, the extension root `./detect` directory is treated as a pre-populated output directory (i.e., extensions can include a static build plan). - Extensions may only output `provides` entries to the build plan. They must not output `requires`. - Extensions are not included in `order` definitions (e.g., in `builder.toml`); instead, a separate `order-extensions` table should be used. The `order-extensions` table will be prepended to each group in the provided `order` (as if `order-extensions` were a composite buildpack). - Extensions are always `optional`. Extensions generate Dockerfiles before the regular buildpack build phase. -To generate these Dockerfiles, the lifecycle executes the extension's `/bin/generate` executable with the same UID, GID, and interface as regular buildpacks. +To generate these Dockerfiles, the lifecycle executes the extension's `./bin/generate` executable with the same UID, GID, and interface as regular buildpacks. However, -- Extensions `/bin/generate` must not write to the app directory. +- Extensions `./bin/generate` must not write to the app directory. - Extensions `` directory is replaced by an `` directory. -- If an extension is missing `/bin/generate`, the extension root `./generate` directory is treated as a pre-populated `` directory. +- If an extension is missing `./bin/generate`, the extension root `./generate` directory is treated as a pre-populated `` directory. -After `/bin/generate` executes, the `` directory may contain +After `./bin/generate` executes, the `` directory may contain - `build.toml`, with the same contents as a normal buildpack's `build.toml` (the `unmet` table array), but - With an additional `args` table array with `name` and `value` fields that are provided as build args to `build.Dockerfile` - `run.toml`, @@ -97,7 +97,7 @@ Support for other instruction formats, e.g., LLB JSON files, could be added in t `build.Dockerfile` and `run.Dockerfile`target the builder image or runtime base image, respectively. -If no Dockerfiles are present, `/bin/generate` may still consume build plan entries. +If no Dockerfiles are present, `./bin/generate` may still consume build plan entries. Dockerfiles are applied to their corresponding base images after all extensions are executed and before any regular buildpacks are executed. Dockerfiles are applied in the order determined during buildpack detection. When multiple Dockerfiles are applied, the intermediate image generated from the application of the current Dockerfile will be provided as the `base_image` ARG to the next Dockerfile. Dockerfiles that target the run image (only) may ignore the provided `base_image` (e.g., `FROM some-other-image`). Dockerfiles that change the runtime base image may still use `COPY --from=${base_image}`. From 9325a131940ad5613bc85b4027b26f44e18da021 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Thu, 14 Jul 2022 15:22:16 -0400 Subject: [PATCH 45/45] Updates per 7/14 Working Group Signed-off-by: Natalie Arellano --- text/0000-dockerfiles.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/text/0000-dockerfiles.md b/text/0000-dockerfiles.md index 23e9eebd7..873a6e3f9 100644 --- a/text/0000-dockerfiles.md +++ b/text/0000-dockerfiles.md @@ -87,8 +87,8 @@ However, - If an extension is missing `./bin/generate`, the extension root `./generate` directory is treated as a pre-populated `` directory. After `./bin/generate` executes, the `` directory may contain -- `build.toml`, with the same contents as a normal buildpack's `build.toml` (the `unmet` table array), but - - With an additional `args` table array with `name` and `value` fields that are provided as build args to `build.Dockerfile` +- `build.toml`, + - With an `args` table array with `name` and `value` fields that are provided as build args to `build.Dockerfile` - `run.toml`, - With an `args` table array with `name` and `value` fields that are provided as build args to `run.Dockerfile` - Either or both of `build.Dockerfile` and `run.Dockerfile` @@ -97,7 +97,7 @@ Support for other instruction formats, e.g., LLB JSON files, could be added in t `build.Dockerfile` and `run.Dockerfile`target the builder image or runtime base image, respectively. -If no Dockerfiles are present, `./bin/generate` may still consume build plan entries. +If no Dockerfiles are present, `./bin/generate` may still consume build plan entries. Unlike buildpacks, extensions must consume all entries in the provided plan (they cannot designate any entries as "unmet"). Dockerfiles are applied to their corresponding base images after all extensions are executed and before any regular buildpacks are executed. Dockerfiles are applied in the order determined during buildpack detection. When multiple Dockerfiles are applied, the intermediate image generated from the application of the current Dockerfile will be provided as the `base_image` ARG to the next Dockerfile. Dockerfiles that target the run image (only) may ignore the provided `base_image` (e.g., `FROM some-other-image`). Dockerfiles that change the runtime base image may still use `COPY --from=${base_image}`. @@ -137,7 +137,6 @@ Note: The Dockerfiles referenced must disable rebasing, and build times will be [ -f package.json ] && cp "$CNB_BUILDPACK_DIR/Dockerfile-node" "$1/build.Dockerfile" ``` - ### Dockerfiles for Base Images The same Dockerfile format may be used to create new base images or modify existing base images outside of the app build process (e.g., before creating a builder). Any specified labels override existing values.