Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Distribution Specification #11

Merged
merged 10 commits into from
Jul 18, 2019
Merged

RFC: Distribution Specification #11

Changes from 3 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
1ec82df
Add distribution spec RFC
sclevine Jun 12, 2019
1c6c93b
Address feedback from WG meeting on 2019-06-12
sclevine Jun 13, 2019
edae953
Small tweaks to distribution spec RFC
sclevine Jun 13, 2019
a2e4627
Dist spec RFC: add buildpack ID to env
sclevine Jun 22, 2019
c4defe5
Specify expansion order and define optional
sclevine Jun 23, 2019
127dfc0
Dist RFC: fix and expand package.toml example + add default key
sclevine Jun 26, 2019
4f3b523
Dist RFC: fix typo, nodejs -> rails
sclevine Jun 26, 2019
255530d
Dist RFC: simplify after WG meeting 2019-06-27
sclevine Jun 27, 2019
d7b1193
Update distribution spec RFC from WG meeting 2019-07-10
sclevine Jul 10, 2019
f1af8b2
Rename 0000-spec-distribution.md to text/0007-spec-distribution.md
sclevine Jul 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
302 changes: 302 additions & 0 deletions 0000-spec-distribution.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,302 @@
# Meta
[meta]: #meta
- Name: Buildpack Distribution Specification
- Start Date: 2019-04-12
- CNB Pull Requests: (spec PR to follow)
- CNB Issues: (lifecycle issues to follow)


# Motivation
[motivation]: #motivation

This proposal enables both decentralized, manual distribution of buildpacks from Docker registries as well as automatic distribution of buildpacks from a centralized buildpack registry.
It allows individual buildpacks to be distributed via a Docker registry, and it makes dynamic assembly of builders on a Docker registry efficient.
It provides a developer-friendly interface that abstracts away complex buildpack configurations.

# What it is
[what-it-is]: #what-it-is

This RFC proposes an official way to distribute buildpacks that conform to the CNB buildpack v3 specification.
Changes would consist of a new Buildpack Distribution specification, modifications to the lifecycle builder, and modifications to the pack CLI.

It affects all personas that interact with buildpacks.

# How it Works
[how-it-works]: #how-it-works

## Overview of Changes

- The `order.toml` and `buildpack.toml` files are merged into `buildpack.toml`.
- `buildpack.toml` can contain multiple buildpacks.
- A buildpack inside of `buildpack.toml` is either:
a. a single buildpack implementation specified by a `path` field or
b. an ordering of other buildpack IDs/versions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we compose buildpackages from each other?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like yes (https://github.com/buildpack/rfcs/pull/11/files#diff-8e4e8df213b3904199aec818ffac0c54R249) unless I'm misunderstanding was "package" here means.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, see the [[packages]] section in package.toml.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So basically have a tree where nodes are either buildpacks or trees of buildpacks...interesting.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

- Every buildpack inside `buildpack.toml` must have an ID and version.
- Multiple buildpack implementations can be defined in `buildpack.toml`, so buildpacks can share files/code.
- A buildpackage is a collection of buildpack repositories with `buildpack.toml` files.
- A buildpackage can have a default buildpack.
- Labels are removed from the specification.
- The lifecycle moves into `/cnb/lifecycle/`.

## Buildpack Blob Format

A buildpack blob is a tgz containing a `buildpack.toml`. Example format:

```toml
[[buildpacks]]
id = "io.buildpacks.nodejs"
name = "Node.js Buildpack"
version = "0.0.9"
[[buildpacks.order]]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

every buildpack in this array must exist in the [[buildpacks]] key?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, they are totally disjoint. E.g., you could fine a repo that just contains a single buildpack with this configuration.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is [[buildpacks.order]] only allowed on any element in the list of [[buildpacks]]?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each buildpack in the top-level list may have a path or an order. The only other place order is valid is builder.toml (but that file is just a config for the pack CLI).

group = [
{ id = "io.buildpacks.node", version = "0.0.5" },
hone marked this conversation as resolved.
Show resolved Hide resolved
{ id = "io.buildpacks.npm", version = "0.0.7" },
]

[[buildpacks]]
id = "io.buildpacks.npm"
name = "NPM Buildpack"
version = "0.0.7"
path = "./npm-cnb/"
[buildpacks.metadata]
# ...
[[buildpacks.stacks]]
id = "io.buildpacks.stacks.bionic"

[[buildpacks]]
id = "io.buildpacks.node"
name = "Node Engine Buildpack"
version = "0.0.5"
path = "./node-cnb/"
[buildpacks.metadata]
# ...
[[buildpacks.stacks]]
id = "io.buildpacks.stacks.bionic"

```

Each `path` must reference a valid buildpack implementation.
However, buildpacks defined in `[[buildpacks.order]]` do not need to be included in the buildpack blob.


## Buildpackage Format

A buildpackage may exist as an OCI image on an image registry, an OCI image in a Docker daemon, or a `.cnb` file.

A `.cnb` file is an uncompressed tar archive containing an OCI image. Its file name should end in `.cnb`.


### Layer Blob

Each FS layer blob in the image contains a single buildpack blob tgz and at least one symlink.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the "single buildpackage blob" or "single tgz containing buildpacks"? Because the example below is confusing given that the checksums for each differant ID is the same. Maybe I'm just misreading here.

Copy link
Member Author

@sclevine sclevine Jun 28, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A blob always means a tgz containing exactly one buildpack.toml and any number of buildpacks (either order definitions or concrete buildpacks).

The checksum is the same in this example, because all buildpacks come from the same blob.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✔️ thanks.

A symlink should be created for each buildpack version that the blob is assembled to support.

```
/cnb/blobs/<sha256 checksum of buildpack blob tgz>/
/cnb/by-id/<buildpack ID1>/<buildpack version> -> /cnb/blobs/<sha256 checksum of blob tgz>/
/cnb/by-id/<buildpack ID2>/<buildpack version> -> /cnb/blobs/<sha256 checksum of blob tgz>/
...
```

Example:

```
/cnb/blobs/bc4c24181ed3ce6666444deeb95e1f61940bffee70dd13972beb331f5d111e9b/
/cnb/by-id/io.buildpacks.nodejs/0.0.9 -> /cnb/blobs/bc4c24181ed3ce6666444deeb95e1f61940bffee70dd13972beb331f5d111e9b/
/cnb/by-id/io.buildpacks.npm/0.0.7 -> /cnb/blobs/bc4c24181ed3ce6666444deeb95e1f61940bffee70dd13972beb331f5d111e9b/
/cnb/by-id/io.buildpacks.node/0.0.5 -> /cnb/blobs/bc4c24181ed3ce6666444deeb95e1f61940bffee70dd13972beb331f5d111e9b/
```

## Buildpackage Metadata

A buildpack ID, buildpack version, and at least one stack must be provided in the OCI image metadata.

Label: `io.buildpacks.cnb.metadata`
```json
{
"id": "io.buildpacks.nodejs",
"version": "0.0.9",
"stacks": [
{
"id": "io.buildpacks.stacks.bionic",
"mixins": ["build:git"]
}
]
}
```

The buildpack ID and version must match a buildpack provided by a layer blob.
For each listed stack, all associated buildpacks must be a candidate for detection when the specified buildpack ID and version are selected.

For a buildpackage to be valid, each entry in `buildpack.toml` must have all listed stacks.
Each stack ID should only be present once, and the `mixins` list should enumerate all the required mixins for that stack to support all included buildpacks.

Fewer stack entries as well as additional mixins for a stack entry may be specified to restrict builders that are created from the buildpackage.

### Execution Order

During detection, a buildpack ID or list of buildpack IDs is resolved into individual buildpack implementation that include a `path` field.

First, the detector determines the user's choice of buildpack IDs or builder's order definition.

Next, the 2-D ordering of buildpacks is derived as follows:

Where:
- O and P are buildpacks IDs referencing buildpacks that compose other buildpack IDs.
- A through H are buildpack IDs referencing executable buildpacks.

Given:

<img src="http://tex.s2cms.ru/svg/%0AO%20%3D%0A%5Cbegin%7Bbmatrix%7D%0AA%2C%20%26%20B%20%5C%5C%0AC%2C%20%26%20D%0A%5Cend%7Bbmatrix%7D%0A" alt="
O =
\begin{bmatrix}
A, &amp; B \\
C, &amp; D
\end{bmatrix}
" />

<img src="http://tex.s2cms.ru/svg/%0AP%20%3D%0A%5Cbegin%7Bbmatrix%7D%0AE%2C%20%26%20F%20%5C%5C%0AG%2C%20%26%20H%0A%5Cend%7Bbmatrix%7D%0A" alt="
P =
\begin{bmatrix}
E, &amp; F \\
G, &amp; H
\end{bmatrix}
" />

We propose:

<img src="http://tex.s2cms.ru/svg/%0A%5Cbegin%7Bbmatrix%7D%0AE%2C%20%26%20O%2C%20%26%20F%0A%5Cend%7Bbmatrix%7D%20%3D%20%0A%5Cbegin%7Bbmatrix%7D%0AE%2C%20%26%20A%2C%20%26%20B%2C%20%26%20F%20%5C%5C%0AE%2C%20%26%20C%2C%20%26%20D%2C%20%26%20F%20%5C%5C%0A%5Cend%7Bbmatrix%7D%0A" alt="
\begin{bmatrix}
E, &amp; O, &amp; F
\end{bmatrix} =
\begin{bmatrix}
E, &amp; A, &amp; B, &amp; F \\
E, &amp; C, &amp; D, &amp; F \\
\end{bmatrix}
" />

<img src="http://tex.s2cms.ru/svg/%0A%5Cbegin%7Bbmatrix%7D%0AO%2C%20%26%20P%0A%5Cend%7Bbmatrix%7D%20%3D%20%0A%5Cbegin%7Bbmatrix%7D%0AA%2C%20%26%20B%2C%20%26%20E%2C%20%26%20F%20%5C%5C%0AA%2C%20%26%20B%2C%20%26%20G%2C%20%26%20H%20%5C%5C%0AC%2C%20%26%20D%2C%20%26%20E%2C%20%26%20F%20%5C%5C%0AC%2C%20%26%20D%2C%20%26%20G%2C%20%26%20H%20%5C%5C%0A%5Cend%7Bbmatrix%7D%0A" alt="
\begin{bmatrix}
O, &amp; P
\end{bmatrix} =
\begin{bmatrix}
A, &amp; B, &amp; E, &amp; F \\
A, &amp; B, &amp; G, &amp; H \\
C, &amp; D, &amp; E, &amp; F \\
C, &amp; D, &amp; G, &amp; H \\
\end{bmatrix}
" />


## User Interface

### App Developer

`pack build` should accept a list of buildpacks via the `--buildpack` flag.
The flag may be passed multiple times to construct a buildpack group.

The value of each flag must be one of:
- A buildpack ID of a buildpack on the builder, optionally followed by `@` and a version
- A path to a buildpackage on the local filesystem
- A reference to a buildpackage on a Docker registry
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In our development workflow this seems non-ideal, because it looks like we lose the ability to directly use a buildpack, because we have to now go:
buildpack-code -> [compiled buildpack] -> [buildpackage].
That makes development of buildpacks a bit more cumbersome, I think.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That’s a good point. Ideally, the whole compilation process could be wrapped into pack create-package, but we would definitely need some kind of pre-package hook in buildpack.toml.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also do local buildpack development this way with --buildpack using pack. There's a huge ease of use factor when being able to use an exploded buildpack directory.


A version must be specified if the version is ambiguous.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when would version be ambiguous?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you have multiple versions of the same buildpack in the builder.


### Buildpack Developer

`pack create-package` will package a selection of

- gzip compressed, tar archived buildpack blobs
- other buildpackages
- stack and buildpack metadata

into a `.cnb` file, OCI image in a registry, or OCI image in a Docker daemon.

These properties will be specified in a `package.toml` file.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file looks very similar to buildpack.toml. Is it crazy to think maybe they should be the same?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

buildpack.toml defines buildpacks -- it has their metadata, names, paths to their source, etc.

package.toml references buildpacks. It's the (potentially ephemeral) configuration file for pack create-package. It contains real-world URIs, etc. that are used to build buildpackages. This is information that can't live in buildpack.toml.


```toml
id = "io.buildpacks.nodejs"
version = "0.0.9"

[[blobs]]
uri = "https://example.com/nodejs.tgz"
[[blobs.buildpacks]]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hone After discussing with @ekcasey, I think it's okay to make the blobs.buildpacks list optional. If unspecified, it means "all buildpacks contained in the blob." This should address the repetition between buildpack.toml and package.toml.

id = "io.buildpacks.nodejs"
version = "0.0.9"
[[blobs.buildpacks]]
id = "io.buildpacks.npm"
version = "0.0.7"
[[blobs.buildpacks]]
id = "io.buildpacks.node"
version = "0.0.5"

[[packages]]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a new key? I couldn't find this in the existing spec.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a new key. There's no existing spec for any of these proposed files outside of this RFC.

Copy link
Contributor

@zmackie zmackie Jun 28, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this key refers to buildpackages, it should say that. If it doesn't I'm confused. I wonder if a more extensible concept like bits (bad name), where each bit says what it is, would allow future types of subpackage more easily. eg:

[[bits]]
uri = "https://example.com/rails.tgz"
[[bits]]
ref = "registry.example.com/nodejs:0.0.9"
[[bits]]
futurething = "some-other-packaging-format.fmt"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I prefer separately lists, because it creates a static schema for the file. I'm not super attached to the names (blobs vs. packages) though.

ref = "registry.example.com/ruby"

[[stacks]]
id = "io.buildpacks.stacks.bionic"
mixins = ["build:git"]
```

`pack create-builder` will generate a builder image from buildpackages, buildpack blobs, and stack metadata.

These properties will be specified in a `builder.toml` file.

```toml
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ekcasey @hone this UX gives builder creators maximum flexibility in defining a builder. Buildpackages are not even necessary.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar feedback to above. This makes creating a builder more complicated (or maybe just confusing) given that buildpackages are now another possible component of the builder. Is the "best practice" to create a buildpackage per the concept formerly known as a label and then compose those or just mix and match buildpackages and buildpacks directly. As a buildpack distributor, I know have to decide that I guess?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A buildpackage is a collection of buildpacks with a single entrypoint ID that references all of those buildpacks (via order definitions). It may work with any number of stacks.

A builder is a single stack image with any number of buildpacks or buildpackages on it.

The builder is useful for building on a workstation, where you need a single copy of the latest buildpacks. It's also useful for building on a platform that doesn't explicitly support CNB (like tekton or concourse).

A buildpackage is more useful for distributing a group of buildpacks that could be used on CF, Heroku, or another buildpack-native platform. It's also a useful way to extend builders that might not have all the buildpacks you need.

In the long term, once we have a centralized buildpack registry that maps buildpack IDs to their package locations, I expect that the builder and package.toml concepts might fade away.

[[blobs]]
uri = "https://example.com/nodejs.tgz"
[[blobs.buildpacks]]
id = "io.buildpacks.nodejs"
version = "0.0.9"
[[blobs.buildpacks]]
id = "io.buildpacks.npm"
version = "0.0.7"
[[blobs.buildpacks]]
id = "io.buildpacks.node"
version = "0.0.5"

[[packages]]
ref = "registry.example.com/ruby"

[[order]]
group = [
{ id = "io.buildpacks.nodejs", version = "0.0.9" },
{ id = "io.buildpacks.ruby", version = "0.1.0" },
]


[stack]
id = "io.buildpacks.stacks.bionic"
mixins = ["build:git"]
build-image = "registry.example.com/build"
run-image = "registry.example.com/run"
run-image-mirrors = ["registry2.example.com/run"]
```

If `order` is not specified, the first buildpackage in `packages` becomes the default buildpack when no buildpacks are specified for `pack build`.

# Unanswered Questions
[questions]: #questions


1. What happens when a resolved ordering of buildpacks has the same ID within a group?
Suggestion: use first, make non-optional if any others are non-optional.

2. Should we allow any buildpack blobs to be present in a buildpackage, regardless of stack?
Suggestion: no, we can define a different format for a large repository of buildpack blobs.

3. For simplicity, should builders be restricted to a single buildpackage, no blobs, and no order definition?
Suggestion: no, the proposed model simplifies the workflow.

4. Should we remove symlinks in a buildpackage to buildpacks that don't match a buildpackage stack?
Suggestion: no, this makes dynamic builder generation difficult.

# Drawbacks
[drawbacks]: #drawbacks

Adding multi-group order definitions together is complex.

# Alternatives
[alternatives]: #alternatives

No competing RFCs are proposed.