From 6402e7fcfbbbbee67a4d2c5d43b8232e92023c78 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Fri, 15 Apr 2022 15:48:33 -0400 Subject: [PATCH] Buildpack api changes to support phase 1 of Dockerfiles implementation Signed-off-by: Natalie Arellano --- buildpack.md | 109 +++++++++++++++++++++++---------------- image_extension.md | 126 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 191 insertions(+), 44 deletions(-) create mode 100644 image_extension.md diff --git a/buildpack.md b/buildpack.md index 38cfef17..aa4f846c 100644 --- a/buildpack.md +++ b/buildpack.md @@ -4,12 +4,13 @@ This document specifies the interface between a lifecycle program and one or mor The lifecycle program uses buildpacks to build software artifacts from source code and pack the result into an OCI image. -This is accomplished in four phases: +This is accomplished in five phases: -1. **Detection,** where an optimal selection of compatible buildpacks is chosen. -2. **Analysis,** where metadata about OCI layers generated during a previous build are made available to buildpacks. -3. **Build,** where buildpacks use that metadata to generate only the OCI layers that need to be replaced. -4. **Export,** where the remote layers are replaced by the generated layers. +1. **Detection**, where an optimal selection of compatible buildpacks is chosen. +1. **Analysis**, where metadata about OCI layers generated during a previous build are made available to buildpacks. +1. **Generation** (optional, see the [Image Extension Interface](image-extension.md)), where image extensions generate Dockerfiles that can be used to select the runtime base image. +1. **Build**, where buildpacks use analyzed metadata to generate only the OCI layers that need to be replaced. +1. **Export**, where the remote layers are replaced by the generated layers. The `ENTRYPOINT` of the OCI image contains logic implemented by the lifecycle that executes during the **Launch** phase. @@ -34,25 +35,29 @@ The `ENTRYPOINT` of the OCI image contains logic implemented by the lifecycle th - [Phase #1: Detection](#phase-1-detection) - [Purpose](#purpose) - [Process](#process) + - [Mixin Satisfaction](#mixin-satisfaction) - [Order Resolution](#order-resolution) - [Phase #2: Analysis](#phase-2-analysis) - [Purpose](#purpose-1) - [Process](#process-1) - - [Phase #3: Build](#phase-3-build) + - [Phase #3: Generation (optional)](#phase-3-generation-optional) - [Purpose](#purpose-2) - [Process](#process-2) + - [Phase #4: Build](#phase-4-build) + - [Purpose](#purpose-3) + - [Process](#process-3) - [Unmet Buildpack Plan Entries](#unmet-buildpack-plan-entries) - - [Software Bill of Materials](#software-bill-of-materials) + - [Software-Bill-of-Materials](#software-bill-of-materials) - [Layers](#layers) - [Providing Layers](#providing-layers) - [Reusing Layers](#reusing-layers) - [Slice Layers](#slice-layers) - - [Phase #4: Export](#phase-4-export) - - [Purpose](#purpose-3) - - [Process](#process-3) - - [Launch](#launch) + - [Phase #5: Export](#phase-5-export) - [Purpose](#purpose-4) - [Process](#process-4) + - [Launch](#launch) + - [Purpose](#purpose-5) + - [Process](#process-5) - [Environment](#environment) - [Provided by the Lifecycle](#provided-by-the-lifecycle) - [Layer Paths](#layer-paths) @@ -79,8 +84,10 @@ The `ENTRYPOINT` of the OCI image contains logic implemented by the lifecycle th - [Order Buildpacks](#order-buildpacks) - [Exec.d Output (TOML)](#execd-output-toml) - [Deprecations](#deprecations) - - [`0.3`](#03) - - [Build Plan (TOML) `requires.version` Key](#build-plan-toml-requiresversion-key) + - [Positional Arguments to `detect` and `build` Executables](#positional-arguments-to-detect-and-build-executables) + - [launch.toml (TOML) `bom` Array](#launchtoml-toml-bom-array) + - [build.toml (TOML) `bom` Array](#buildtoml-toml-bom-array) + - [Build Plan (TOML) `requires.version` Key](#build-plan-toml-requiresversion-key) ## Buildpack API Version This document specifies Buildpack API version `0.8` @@ -134,7 +141,6 @@ Executable: `/bin/detect`, Working Dir: `` | Standard error | Logs (warnings, errors) | | `$CNB_BUILD_PLAN_PATH` | Contributions to the the Build Plan (TOML) | - ### Build Executable: `/bin/build`, Working Dir: `` @@ -306,36 +312,37 @@ At the end of each individual buildpack's build phase: ### Purpose -The purpose of detection is to find an ordered group of buildpacks to use during the build phase. -These buildpacks must be compatible with the app. +The purpose of detection is to find an ordered group of image extensions and executable buildpacks to use during the generation and build phases. + +For detect requirements that are specific to image extensions, see the [Image Extension Interface](image-extension.md). ### Process **GIVEN:** -- An ordered list of buildpack groups resolved into buildpack implementations as described in [Order Resolution](#order-resolution) +- An ordered list of resolved groups as described in [Order Resolution](#order-resolution) - A directory containing application source code - A shell, if needed, -For each buildpack in each group in order, the lifecycle MUST execute `/bin/detect`. +For each image extension ("extension") or executable buildpack ("buildpack") in each group in order, the lifecycle MUST execute `/bin/detect`. Image extensions MUST always be optional during detection. -1. **If** the exit status of `/bin/detect` is non-zero and the buildpack is not marked optional, \ +1. **If** the exit status of a buildpack `/bin/detect` is non-zero and the buildpack is not marked optional, \ **Then** the lifecycle MUST proceed to the next group or fail detection completely if no more groups are present. -2. **If** the exit status of `/bin/detect` is zero or the buildpack is marked optional, - 1. **If** the buildpack is not the last buildpack in the group, \ - **Then** the lifecycle MUST proceed to the next buildpack in the group. +2. **If** the exit status of `/bin/detect` is zero or the extension or buildpack is marked optional, + 1. **If** the extension or buildpack is not the last in the group, \ + **Then** the lifecycle MUST proceed to the next extension or buildpack in the group. - 2. **If** the buildpack is the last buildpack in the group, - 1. **If** no exit statuses from `/bin/detect` in the group are zero \ + 2. **If** the extension or buildpack is the last in the group, + 1. **If** no exit statuses from buildpack `/bin/detect` in the group are zero \ **Then** the lifecycle MUST proceed to the next group or fail detection completely if no more groups are present. - 2. **If** at least one exit status from `/bin/detect` in the group is zero \ - **Then** the lifecycle MUST select this group and proceed to the analysis phase. + 2. **If** at least one exit status from a buildpack `/bin/detect` in the group is zero \ + **Then** the lifecycle MUST select this group and MAY proceed to the generation phase. -The selected group MUST be filtered to only include buildpacks with exit status zero. -The order of the buildpacks in the group MUST otherwise be preserved. +The selected group MUST be filtered to only include extensions and buildpacks with exit status zero. +The order of the extensions and buildpacks in the group MUST otherwise be preserved. -The `/bin/detect` executable in each buildpack, when executed: +The `/bin/detect` executable in each extension or buildpack, when executed: - MAY read the app directory. - MAY read the detect environment as described in the [Environment](#environment) section. @@ -349,23 +356,24 @@ Each `requires` and `provides` section MUST be a list of entries formatted as de Each pairing of `requires` and `provides` sections (at the top level, or inside of an `or` array) is a potential Build Plan. -For a given buildpack group, a sequence of trials is generated by selecting a single potential Build Plan from each buildpack in a left-to-right, depth-first order. +For a given group, a sequence of trials is generated by selecting a single potential Build Plan from each extension or buildpack in a left-to-right, depth-first order. The group fails to detect if all trials fail to detect. For each trial, - If a required buildpack provides a dependency that is not required by the same buildpack or a subsequent buildpack, the trial MUST fail to detect. - If a required buildpack requires a dependency that is not provided by the same buildpack or a previous buildpack, the trial MUST fail to detect. - If an optional buildpack provides a dependency that is not required by the same buildpack or a subsequent buildpack, the optional buildpack MUST be excluded from the build phase and its requires and provides MUST be excluded from the Build Plan. +- If an extension provides a dependency that is not required by a subsequent buildpack, the extension MUST be excluded from the build phase and its provides MUST be excluded from the Build Plan. - If an optional buildpack requires a dependency that is not provided by the same buildpack or a previous buildpack, the optional buildpack MUST be excluded from the build phase and its requires and provides MUST be excluded from the Build Plan. - Multiple buildpacks MAY require or provide the same dependency. +- Multiple extensions MAY provide the same dependency. The lifecycle MAY execute each `/bin/detect` within a group in parallel. -The lifecycle MUST run `/bin/detect` for all buildpacks in a group in a container using common stack with a common set of mixins. -The lifecycle MUST fail detection if any of those buildpacks does not list that stack in `buildpack.toml`. -The lifecycle MUST fail detection if any of those buildpacks specifies a mixin associated with that stack in `buildpack.toml` that is not satisfied, see [Mixin Satisfaction](#mixin-satisfaction) below. +The lifecycle MUST run `/bin/detect` for all extensions and buildpacks in a group in a container using a common build base image. #### Mixin Satisfaction + A buildpack's mixin requirements must be satisfied by the stack in one of the following scenarios. 1) the stack provides the mixin `run:` and the buildpack requires `run:` @@ -378,13 +386,17 @@ A buildpack's mixin requirements must be satisfied by the stack in one of the fo #### Order Resolution -During detection, an order definition MUST be resolved into individual buildpack implementations. +During detection, an order definition for image extensions (if present) and an order definition for buildpacks MUST be resolved into a group of image extensions and executable buildpacks. + +Order definitions for image extensions MUST NOT contain nested orders. + +If an order definition for image extensions is present, it MUST be prepended to the order definition for buildpacks before the resolution process occurs. The resolution process MUST follow this pattern: Where: -- O and P are buildpack orders. -- A through H are buildpack implementations. +- O and P are image extension or buildpack orders. +- A through H are image extensions or executable buildpacks. Given: @@ -461,7 +473,17 @@ For each buildpack in the group, the lifecycle After analysis, the lifecycle MUST proceed to the build phase. -## Phase #3: Build +## Phase #3: Generation (optional) + +### Purpose + +The purpose of the generation phase is to generate Dockerfiles that can be used to select the runtime base image. + +### Process + +See the [Image Extension Specification](#image-extension.md). + +## Phase #4: Build ![Build](img/build.svg) @@ -549,14 +571,14 @@ Correspondingly, each `/bin/build` executable: #### Unmet Buildpack Plan Entries -A buildpack SHOULD designate a Buildpack Plan entry as unmet if the buildpack did not satisfy the requirement described by the entry. +An image extension or executable buildpack SHOULD designate a Buildpack Plan entry as unmet if it did not satisfy the requirement described by the entry. The lifecycle SHALL assume that all entries in the Buildpack Plan were satisfied by the buildpack unless the buildpack writes an entry with the given name to the `unmet` section of `build.toml`. For each entry in ``: - **If** there is an unmet entry in `build.toml` with a matching `name`, the lifecycle - - MUST include the entry in the `` of the next buildpack that provided an entry with that name during the detection phase. + - MUST include the entry in the `` of the next image extension or executable buildpack that provided an entry with that name during the detection phase. - **Else**, the lifecycle - - MUST NOT include entries with matching names in the `` provided to subsequent buildpacks. + - MUST NOT include entries with matching names in the `` provided to subsequent image extensions or executable buildpacks. #### Software-Bill-of-Materials @@ -612,7 +634,7 @@ Additionally, a buildpack MAY specify sub-paths within `` as `slices` in `l Separate layers MUST be created during the export phase for each slice with one or more files or directories. This minimizes data transfer when the app directory contains a known set of files. -## Phase #4: Export +## Phase #5: Export ![Export](img/export.svg) @@ -877,7 +899,7 @@ Prohibited: ### Requirements -The lifecycle MUST be implemented so that the detection and build phases do not have access to OCI image store credentials used in the analysis and export phases. +The lifecycle MUST be implemented so that the detection, generation, and build phases do not have access to OCI image store credentials used in the analysis and export phases. The lifecycle SHOULD be implemented so that each phase may run in a different container. ## Data Format @@ -1099,7 +1121,7 @@ Each stack in `stacks` either: #### Order Buildpacks -A buildpack descriptor that specifies `order` MUST be [resolvable](#order-resolution) into an ordering of buildpacks that implement the [Buildpack Interface](#buildpack-interface). +A buildpack that specifies `order` in its `buildpack.toml` MUST be [resolvable](#order-resolution) into an ordering of executable buildpacks that implement the [Buildpack Interface](#buildpack-interface). The `order` MUST include only executable buildpacks and MUST NOT include image extensions. A buildpack reference inside of a `group` MUST contain an `id` and `version`. @@ -1120,7 +1142,6 @@ Each `key`: ## Deprecations This section describes all the features that are deprecated. - ### Positional Arguments to `detect` and `build` Executables _Deprecated in Buildpack API 0.8._ diff --git a/image_extension.md b/image_extension.md new file mode 100644 index 00000000..c412c9db --- /dev/null +++ b/image_extension.md @@ -0,0 +1,126 @@ +# Image Extension Interface Specification + +This document specifies the interface between a lifecycle program and one or more image extensions. + +The lifecycle program uses image extensions to generate Dockerfiles that can be used to select the runtime base image prior to executable buildpack builds. + +## Table of Contents + + +- [Image Extension Interface Specification](#image-extension-interface-specification) + - [Table of Contents](#table-of-contents) + - [Image Extension API Version](#image-extension-api-version) + - [Image Extension Interface](#image-extension-interface) + - [Detection](#detection) + - [Generation](#generation) + - [Phase: Generation](#phase-generation) + - [Purpose](#purpose) + - [Process](#process) + - [Dockerfile Requirements](#dockerfile-requirements) + - [Data Format](#data-format) + - [Files](#files) + - [extension.toml (TOML)](#extensiontoml-toml) + +## Image Extension API Version + +This document accompanies Buildpack API version `0.9`. + +## Image Extension Interface + +Unless otherwise noted, image extensions are expected to conform to the [Buildpack Interface Specification](buildpack.md). + +### Detection + +Executable: `/bin/detect`, Working Dir: `` + +Image extensions participate in the buildpack [detection](buildpack.md#detection) process, with the same interface for `/bin/detect`. However: +- Detection is optional for image extensions, and they are assumed to pass detection when `/bin/detect` is not present. +- Image extensions MUST only output `provides` entries to the build plan. They MUST NOT output `requires`. + +### Generation + +Executable: `/bin/generate`, Working Dir: `` + +Image extensions participate in a generation process that is similar to the buildpack [build](buildpack.md#build) process, with an interface that is similar to `/bin/build`. However: +- Image extensions' `/bin/generate` MUST NOT write to the app directory. +- Instead of the `CNB_LAYERS_DIR`, image extensions receive a `CNB_OUTPUT_DIR` which MUST be the absolute path of an `` directory and MUST NOT be the path of the buildpack layers directory. +- If an image extension is missing `/bin/generate`, the image extension root MUST be treated as a pre-populated `` directory. + +## Phase: Generation + +### Purpose + +The purpose of the generation phase is to generate Dockerfiles that can be used to select the runtime base image. + +### Process + +**GIVEN:** +- The final ordered group of image extensions determined during the detection phase, +- A directory containing application source code, +- The Buildpack Plan, +- An `` directory used to store generated artifacts, +- A shell, if needed, + +For each image extension in the group in order, the lifecycle MUST execute `/bin/generate`. + +1. **If** the exit status of `/bin/generate` is non-zero, \ + **Then** the lifecycle MUST fail the build. + +2. **If** the exit status of `/bin/generate` is zero, + 1. **If** there are additional image extensions in the group, \ + **Then** the lifecycle MUST proceed to the next image extension's `/bin/generate`. + + 2. **If** there are no additional image extensions in the group, \ + **Then** the lifecycle MUST proceed to the build phase. + +For each `/bin/generate` executable in each image extension, the lifecycle: + +- MUST provide path arguments to `/bin/generate` as described in the [generation](#generation) section. +- MUST configure the build environment as described in the [Environment](buildpack.md#environment) section. +- MUST provide all `` entries that were required by any buildpack in the group during the detection phase with names matching the names that the image extension provided. + +Correspondingly, each `/bin/generate` executable: + +- MAY read from the `` directory. +- MUST NOT write to the `` directory. +- MAY read the build environment as described in the [Environment](buildpack.md#environment) section. +- MAY read the Buildpack Plan. +- MAY log output from the build process to `stdout`. +- MAY emit error, warning, or debug messages to `stderr`. +- MAY write a run.Dockerfile to the `` directory. This file MUST adhere to the requirements listed below. +- MUST NOT write SBOM (Software-Bill-of-Materials) files as described in the [Software-Bill-of-Materials](#software-bill-of-materials) section. + +#### Dockerfile Requirements + +run.Dockerfiles: + +- MAY contain a single `FROM` instruction +- MUST NOT contain any other instructions + +## Data Format + +### Files + +### extension.toml (TOML) + +This section describes the 'Extension descriptor'. + +```toml +api = "" + +[extension] +id = "" +name = "" +version = "" +homepage = "" +description = "" +keywords = [ "" ] + +[[extension.licenses]] +type = "" +uri = "" +``` + +Image extension authors MUST choose a globally unique ID, for example: "io.buildpacks.apt". + +The image extension `id`, `version`, `api`, `licenses`, and `sbom-formats` entries MUST follow the requirements defined in the [Buildpack Interface Specification](buildpack.md).