RFC: Distribution Specification

[sclevine]

Readable.

[sclevine]

@hone @jkutner @nebhale @ekcasey @matthewmcnew
Addressed feedback from the WG meeting today. Now includes package.toml, builder.toml, and addresses usability feedback.

[sclevine]

@ekcasey @hone this UX gives builder creators maximum flexibility in defining a builder. Buildpackages are not even necessary.

[zmackie]

Similar feedback to above. This makes creating a builder more complicated (or maybe just confusing) given that buildpackages are now another possible component of the builder. Is the "best practice" to create a buildpackage per the concept formerly known as a label and then compose those or just mix and match buildpackages and buildpacks directly. As a buildpack distributor, I know have to decide that I guess?

[sclevine]

A buildpackage is a collection of buildpacks with a single entrypoint ID that references all of those buildpacks (via order definitions). It may work with any number of stacks.

A builder is a single stack image with any number of buildpacks or buildpackages on it.

The builder is useful for building on a workstation, where you need a single copy of the latest buildpacks. It's also useful for building on a platform that doesn't explicitly support CNB (like tekton or concourse).

A buildpackage is more useful for distributing a group of buildpacks that could be used on CF, Heroku, or another buildpack-native platform. It's also a useful way to extend builders that might not have all the buildpacks you need.

In the long term, once we have a centralized buildpack registry that maps buildpack IDs to their package locations, I expect that the builder and package.toml concepts might fade away.

[sclevine]

@jkutner I added the buildpack ID to the environment as $BP_ID. @nebhale pointed out that otherwise, buildpacks would need to hardcode their ID to figure out which entry they correspond to inbuildpack.toml.

[sclevine]

Two things I realized need to be specified while playing with this API over the weekend:

1. The definition of optional on an order definition buildpack is that the lifecycle should repeat the group it is specified in without the order definition. If we naively assigned optional to all sub-buildpacks in the order definition, we could break nested order definitions that rely on some buildpacks not being optional.

   This is completely consistent with the current meaning of optional for a concrete buildpack implementation. The current behavior is really just a performance optimization on a “leaf node” that requires fewer re-tries.

2. We should define the expansion as depth-first in left-to-right over. This should work well with add-on type buildpacks that instrument a built app. It’s also easiest to understand (I think).

[hone]

every buildpack in this array must exist in the [[buildpacks]] key?

[sclevine]

Nope, they are totally disjoint. E.g., you could fine a repo that just contains a single buildpack with this configuration.

[jkutner]

is [[buildpacks.order]] only allowed on any element in the list of [[buildpacks]]?

[sclevine]

Each buildpack in the top-level list may have a path or an order. The only other place order is valid is builder.toml (but that file is just a config for the pack CLI).

[hone]

when would version be ambiguous?

[sclevine]

If you have multiple versions of the same buildpack in the builder.

[hone]

This file looks very similar to buildpack.toml. Is it crazy to think maybe they should be the same?

[sclevine]

buildpack.toml defines buildpacks -- it has their metadata, names, paths to their source, etc.

package.toml references buildpacks. It's the (potentially ephemeral) configuration file for pack create-package. It contains real-world URIs, etc. that are used to build buildpackages. This is information that can't live in buildpack.toml.

[hone]

Is this a new key? I couldn't find this in the existing spec.

[sclevine]

This is a new key. There's no existing spec for any of these proposed files outside of this RFC.

[zmackie]

If this key refers to buildpackages, it should say that. If it doesn't I'm confused. I wonder if a more extensible concept like bits (bad name), where each bit says what it is, would allow future types of subpackage more easily. eg:

[[bits]]
uri = "https://example.com/rails.tgz"
[[bits]]
ref = "registry.example.com/nodejs:0.0.9"
[[bits]]
futurething = "some-other-packaging-format.fmt"

[sclevine]

I think I prefer separately lists, because it creates a static schema for the file. I'm not super attached to the names (blobs vs. packages) though.

[sclevine]

@hone After discussing with @ekcasey, I think it's okay to make the blobs.buildpacks list optional. If unspecified, it means "all buildpacks contained in the blob." This should address the repetition between buildpack.toml and package.toml.

[ameyer-pivotal]

I really like this. One non-trivial concern I have is with the naming "buildpackage". Since, "pack" is often short for "package", I think it will lead to confusion between "buildpack" and "buildpackage". Instead, I'd like to propose "buildpack bundle" (or something similar, if anyone has ideas).

[sclevine]

It was called a bundle in the original proposal, but @ekcasey suggested buildpackage instead. I don't have a strong opinion.

[zmackie]

In our development workflow this seems non-ideal, because it looks like we lose the ability to directly use a buildpack, because we have to now go:
buildpack-code -> [compiled buildpack] -> [buildpackage].
That makes development of buildpacks a bit more cumbersome, I think.

[sclevine]

That’s a good point. Ideally, the whole compilation process could be wrapped into pack create-package, but we would definitely need some kind of pre-package hook in buildpack.toml.

[hone]

I also do local buildpack development this way with --buildpack using pack. There's a huge ease of use factor when being able to use an exploded buildpack directory.

[zmackie]

Can we compose buildpackages from each other?

[zmackie]

It looks like yes (https://github.com/buildpack/rfcs/pull/11/files#diff-8e4e8df213b3904199aec818ffac0c54R249) unless I'm misunderstanding was "package" here means.

[sclevine]

Yep, see the [[packages]] section in package.toml.

[zmackie]

So basically have a tree where nodes are either buildpacks or trees of buildpacks...interesting.

[sclevine]

👍

[zmackie]

is the "single buildpackage blob" or "single tgz containing buildpacks"? Because the example below is confusing given that the checksums for each differant ID is the same. Maybe I'm just misreading here.

[sclevine]

A blob always means a tgz containing exactly one buildpack.toml and any number of buildpacks (either order definitions or concrete buildpacks).

The checksum is the same in this example, because all buildpacks come from the same blob.

[zmackie]

✔️ thanks.

[zmackie]

Some feedback from the perspective of release engineering buildpacks for distribution. Overall in support, but I think this complicates our distribution and development story. Which is okay if we feel that this ultimately eases pains of consuming buildpacks. I missed the WG meeting, but is a registry somehow a better distribution point than a uri pointing to a tarball b/c of content adressability or something like that?

[sclevine]

Using a docker registry has a few key benefits:

1. Builders and buildpackages can be constructed dynamically on a registry without (significant) data transfer.
2. Only the changed blobs are downloaded when you download a new copy of a buildpackage or builder from a registry.

In the long term, a centralized buildpack registry might construct a buildpackage on-the-fly when you pull it by ID, e.g.,

docker pull bp.example.com/org.cloudfoundry.ruby

Could dynamically feed you a manifest that references the buildpack blobs contained by that ID.

[sclevine]

@jkutner I updated this to support the workflows you brought up today. Both pack build --buildpack <blob-dir> and pack create-package <blob-dir> are now supported.

I used the first buildpack in buildpack.toml for now, but happy to change as needed.

[jkutner]

I was hoping we could arrive at something like this (I don't like having buildpack & buildpacks in the schema but ignore that):

[buildpack]
id = "io.buildpacks.nodejs"
name = "Node.js Buildpack"
version = "0.0.9"
[[buildpack.order]]
group = [
   { id = "io.buildpacks.node", version = "0.0.5" },
   { id = "io.buildpacks.npm", version = "0.0.7" },
]

[[buildpacks]]
id = "io.buildpacks.npm"
name = "NPM Buildpack"
version = "0.0.7"
path = "./npm-cnb/"
[buildpacks.metadata]
# ...
[[buildpacks.stacks]]
id = "io.buildpacks.stacks.bionic"

[[buildpacks]]
id = "io.buildpacks.node"
name = "Node Engine Buildpack"
version = "0.0.5"
path = "./node-cnb/"
[buildpacks.metadata]
# ...
[[buildpacks.stacks]]
id = "io.buildpacks.stacks.bionic"

The idea being that every buildpack.toml has a single buildpack that is akin to what you've defined as the "first buildpack".

But given that we didn't land on something like ^^, I think the file should be named buildpacks.toml. I'm happy to open that in its own RFC though.

I'm going to approve on this, but I still think that there may be a better way to capture the "first buildpack" as being special--in a way that is more obvious to the author/reader of the buildpack.toml.

[sclevine]

Thanks @jkutner. Hopefully going with the first buildpack for now will make it easy to override that behavior with a different mechanism once we identify it. Happy to consider the current solution a starting point, and works towards functionality that makes dealing buildpack blobs directly an ideal experience.

Waiting for @hone to approve before merging.

[sclevine]

@hone reminder about this RFC

[hone]

@jkutner is buildpacks.toml redundant if everything is a buildpack? :) but I agree with the sentiment. Just to iterate, I think most people who are looking at buildpacks in the Heroku ecosystem and starting out are probably looking at the single (primary) buildpack and everything else is built to support it.

There were concerns brought up about mixing defining metadata and how to package, but from I can tell this is common in most language ecosystems in dependency managers like package.json or Cargo.toml that includes both metadata about the package and how to actually build them.

Let's move forward and see how it feels in practice and revisit when the registry is around.

[sclevine]

There were concerns brought up about mixing defining metadata and how to package, but from I can tell this is common in most language ecosystems in dependency managers like package.json or Cargo.toml that includes both metadata about the package and how to actually build them.

Just to clarify, I'm only concerned about putting a mandatory single URI for each buildpack reference in buildpack.toml. That's the only function I see package.toml providing right now. Once we have a registry, I'm very supportive of it going away, and of us adding a registries key to buildpack.toml.