From 9e63ee3f39590a7e18b77985f2541c04669ecf1f Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 6 Mar 2023 16:21:33 -0500 Subject: [PATCH 1/9] WIP - initial version for export to OCI layout documentation Signed-off-by: Juan Bustamante --- .../export-to-oci-layout.md | 106 ++++++++++++++++++ 1 file changed, 106 insertions(+) create mode 100644 content/docs/app-developer-guide/export-to-oci-layout.md diff --git a/content/docs/app-developer-guide/export-to-oci-layout.md b/content/docs/app-developer-guide/export-to-oci-layout.md new file mode 100644 index 000000000..2503a43fe --- /dev/null +++ b/content/docs/app-developer-guide/export-to-oci-layout.md @@ -0,0 +1,106 @@ ++++ +title="Export to OCI layout format" +weight=3 +summary="Learn how to export your application image to disk in OCI layout format" ++++ + +
+ The OCI Image Layout is the directory structure for OCI content-addressable blobs and location-addressable references. +
See the specification
+
+ +Exporting to OCI layout format is an **experimental** feature available on pack since version X.Y.Z + +### 1. Enable experimental feature + +Verify your pack version is equal or greater than X.Y.Z + +```bash +pack version +``` + +Enable the experimental features on pack + +```bash +pack config experimental true +``` + +You can confirm everything is fine, checking the `config.toml` file in your `PACK_HOME` installation folder, for example: + +```bash +cat ~/.pack/config.toml +experimental = true +layout-repo-dir = "<$HOME>/.pack/layout-repo" +``` + +The configuration shows the experimental mode was **enabled** and a local directory to save images on disk was configured to path `<$HOME>/.pack/layout-repo`. + +### 2. Build the app + +If you haven't already, please follow the steps to [build an app](/docs/app-developer-guide/build-an-app). + +The OCI layout feature must be enabled using the convention `oci:` in the `` parameter when invoking `pack build`. + +For example: + +```bash +pack build oci:sample-app --path samples/apps/java-maven --builder cnbs/sample-builder:bionic +``` + +It will save the image in a folder *sample-app* created in the current directory. + +### 3. Check your image + +**Congratulations!** + +You can verify your application image was saved on disk in a folder called *sample-app* in your current directory in OCI layout format, for example: + +```bash +tree sample-app + +sample-app +├── blobs +│ └── sha256 +│ ├── 2fa192256ce255c6ea6c1296eadfe2feba8094f40e6aa85e699645caca2e85d8 +│ ├── 5a44e4f7b58d74fe6f92dd7028075c91191128d1e2e7f39846fe061a9a98836e +│ └── 622426666a7b61c086c84203082d5f64495be1f8b085137e53b0554cfcdb50ab +├── index.json +└── oci-layout +``` + +--- + +## Extra configuration + +### Skip saving your run-image layers on disk + +If you don't need your `run-image` layers on disk, you can skip them using `--sparse` flag in your `pack build` command invocation + +## Implementation notes + +### Media Types + +According to the OCI specification, the [compatibles media types](https://github.com/opencontainers/image-spec/blob/main/media-types.md#compatibility-matrix) for the index.json files must be: + +- `application/vnd.oci.image.index.v1+json` or +- `application/vnd.docker.distribution.manifest.list.v2+json` + +If you are trying to use the lifecycle directly without using `pack` to export your image, take on consideration that tools like: + +[skopeo](https://github.com/containers/skopeo) +```bash +skopeo copy -a docker:// +``` +It will give you `application/vnd.oci.image.index.v1+json` media type, which is currently working + +But [crane](https://github.com/google/go-containerregistry/tree/main/cmd/crane) + +```bash +crane pull --format=oci +``` +It will give you `application/vnd.docker.distribution.manifest.list.v2+json`, which will fail + + + + + From 24038b93992b2c443d4de8750f65593a13ea94e9 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 6 Mar 2023 12:40:56 -0500 Subject: [PATCH 2/9] fixing hugo download issue Signed-off-by: Juan Bustamante --- Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 7922a91ac..4e19bb145 100644 --- a/Makefile +++ b/Makefile @@ -43,7 +43,7 @@ endif endif ifeq ($(shell uname -s),Darwin) - HUGO_OS:=macOS + HUGO_OS:=darwin HUGO_ARCH:=universal endif @@ -188,4 +188,4 @@ katacoda: check-katacoda: katacoda @echo "Checking if Katacoda docs are up-to-date..." @git diff --quiet HEAD -- katacoda/scenarios || ( echo "Katacoda docs are not up-to-date! Please run 'make katacoda' and commit the katacoda/scenarios folder" && exit 1) - @echo "All katacoda docs are up-to-date" \ No newline at end of file + @echo "All katacoda docs are up-to-date" From 0b0121aa946e354e9a8303100a9d2b233540bd99 Mon Sep 17 00:00:00 2001 From: Tim Downey Date: Fri, 24 Feb 2023 14:50:27 -0700 Subject: [PATCH 3/9] Update Gemfile in buildpack author guide - Ruby 3+ no longer includes webrick so it has to be explicitly required in the Gemfile for the app to start - See: https://bugs.ruby-lang.org/issues/17303 Signed-off-by: Tim Downey --- .../create-buildpack/setup-local-environment.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/content/docs/buildpack-author-guide/create-buildpack/setup-local-environment.md b/content/docs/buildpack-author-guide/create-buildpack/setup-local-environment.md index f0154253f..27250f7d9 100644 --- a/content/docs/buildpack-author-guide/create-buildpack/setup-local-environment.md +++ b/content/docs/buildpack-author-guide/create-buildpack/setup-local-environment.md @@ -46,11 +46,12 @@ Then, create a file called `ruby-sample-app/Gemfile` with the ```ruby -source "https://rubygems.org" +source 'https://rubygems.org' git_source(:github) {|repo_name| "https://github.com/#{repo_name}" } -gem "sinatra" +gem 'sinatra' +gem 'webrick' ``` Finally, make sure your local Docker daemon is running by executing: From 3a9c147e52b48b6e8c55cab5414d8663ad4e55ab Mon Sep 17 00:00:00 2001 From: Philipp Stehle Date: Fri, 17 Mar 2023 17:18:59 +0100 Subject: [PATCH 4/9] Replace deprecated GHA ::set-output syntax Signed-off-by: Philipp Stehle --- .../publishing-with-github-actions.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/docs/buildpack-author-guide/publishing-with-github-actions.md b/content/docs/buildpack-author-guide/publishing-with-github-actions.md index 4d7ab94c9..a3fb69ef2 100644 --- a/content/docs/buildpack-author-guide/publishing-with-github-actions.md +++ b/content/docs/buildpack-author-guide/publishing-with-github-actions.md @@ -52,9 +52,9 @@ jobs: PACKAGE="${REPO}/$(echo "$BP_ID" | sed 's/\//_/g')" pack buildpack package --publish ${PACKAGE}:${VERSION} DIGEST="$(crane digest ${PACKAGE}:${VERSION})" - echo "::set-output name=bp_id::$BP_ID" - echo "::set-output name=version::$VERSION" - echo "::set-output name=address::${PACKAGE}@${DIGEST}" + echo "bp_id=$BP_ID" >> "$GITHUB_OUTPUT" + echo "version=$VERSION" >> "$GITHUB_OUTPUT" + echo "address=${PACKAGE}@${DIGEST}" >> "$GITHUB_OUTPUT" shell: bash env: REPO: docker.io/${{ secrets.DOCKER_HUB_USER }} From 56c077a21363c7da74668d0ff569526847c9bd6b Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 27 Mar 2023 17:07:14 -0500 Subject: [PATCH 5/9] adding more detail to undraft the documentation Signed-off-by: Juan Bustamante --- .../export-to-oci-layout.md | 63 ++++++++++++++++--- 1 file changed, 54 insertions(+), 9 deletions(-) diff --git a/content/docs/app-developer-guide/export-to-oci-layout.md b/content/docs/app-developer-guide/export-to-oci-layout.md index 2503a43fe..4edf6bf32 100644 --- a/content/docs/app-developer-guide/export-to-oci-layout.md +++ b/content/docs/app-developer-guide/export-to-oci-layout.md @@ -37,7 +37,7 @@ The configuration shows the experimental mode was **enabled** and a local direct ### 2. Build the app -If you haven't already, please follow the steps to [build an app](/docs/app-developer-guide/build-an-app). +If you haven't already, please follow the steps to [build an app](/docs/app-developer-guide/build-an-app), once you managed to build a sample application you can try exporting the sample application to disk in OCI layout format. The OCI layout feature must be enabled using the convention `oci:` in the `` parameter when invoking `pack build`. @@ -47,26 +47,38 @@ For example: pack build oci:sample-app --path samples/apps/java-maven --builder cnbs/sample-builder:bionic ``` -It will save the image in a folder *sample-app* created in the current directory. +It will save the image in a folder `./sample-app` created in your current directory. ### 3. Check your image **Congratulations!** -You can verify your application image was saved on disk in a folder called *sample-app* in your current directory in OCI layout format, for example: +You can verify your application image was saved on disk in a folder called `./sample-app` in your current directory in OCI layout format, for example: ```bash tree sample-app sample-app ├── blobs -│ └── sha256 -│ ├── 2fa192256ce255c6ea6c1296eadfe2feba8094f40e6aa85e699645caca2e85d8 -│ ├── 5a44e4f7b58d74fe6f92dd7028075c91191128d1e2e7f39846fe061a9a98836e -│ └── 622426666a7b61c086c84203082d5f64495be1f8b085137e53b0554cfcdb50ab +│ └── sha256 +│ ├── 141bfb0cd434d425bc70edb9e56ea11d07aed76450eb0e73e6110645f251a8d3 +│ ├── 2fa192256ce255c6ea6c1296eadfe2feba8094f40e6aa85e699645caca2e85d8 +│ ├── 5a44e4f7b58d74fe6f92dd7028075c91191128d1e2e7f39846fe061a9a98836e +│ ├── 72d9f18d70f395ff9bfae4d193077ccea3ca583e3da3dd66f5c84520c0100727 +│ ├── 827746ec7ba80f4e4811b6c9195b6f810fbc2d58a6c9cc337bf0305791f24e97 +│ ├── ad13830c92258c952f25d561d8bf7d9eb58b8a3003960db1502cbda8239130b5 +│ ├── b97b58b190d5f731c879b0f7446a2bd554863b51851e03757199c74dd922ce61 +│ ├── c44222730efa142cd5bedc0babf82a9a07d325494be7f5c3cfde56f43166b65f +│ ├── e1048fb89c3194a1f0542c0847aa086a7034dd7867c48fe8c93675cf36f90610 +│ ├── f0a30c5bc44742065b1b4ffa95271a39994f05ba7a03dd7e7143d1d3e45fa0b1 +│ └── f9d6350d0c44c0e7165a522155f53181ce8c163a6b8ead1f6baea22d1a8d8a78 ├── index.json -└── oci-layout +└── oci-layout + +3 directories, 13 files ``` +If you are interested on keep playing with the image in OCI layout format, one tool you can take a look is [umoci](https://umo.ci/) it can help you to create a +[runtime bundler](https://github.com/opencontainers/runtime-spec) that can be executed with another tool like [runc](https://github.com/opencontainers/runc) --- @@ -74,7 +86,40 @@ sample-app ### Skip saving your run-image layers on disk -If you don't need your `run-image` layers on disk, you can skip them using `--sparse` flag in your `pack build` command invocation +Before using this option we suggest to remove your local layout directory (the one configured in your pack config.toml with the key `layout-repo-dir`) and +your application image folder (if you are planning to use the same folder), the reason for this is pack doesn't remove the blobs saved in the `layout-repo-dir` if you use the `--sparse` flag + +If you don't need your `run-image` layers on disk, you can skip them using `--sparse` flag in your `pack build` command invocation. + +For example: + +```bash +pack build oci:sample-app --sparse --path samples/apps/java-maven --builder cnbs/sample-builder:bionic +``` + +Verify your application image + +```bash +sample-app +├── blobs +│ └── sha256 +│ ├── 2ebed3ab57806441e2bf814eaf0648ed77289e058340d2b76d32b422fbaac5d8 +│ ├── 2fa192256ce255c6ea6c1296eadfe2feba8094f40e6aa85e699645caca2e85d8 +│ ├── 5a44e4f7b58d74fe6f92dd7028075c91191128d1e2e7f39846fe061a9a98836e +│ ├── 741e558b7b807fea350b26b8152170a2463277cb3d1268b60de76ec12608518a +│ ├── 907c84671180d979a38affb62d9a6ea8e9a510e27639e0b60a34a42f1a846ddc +│ ├── ad13830c92258c952f25d561d8bf7d9eb58b8a3003960db1502cbda8239130b5 +│ ├── c44222730efa142cd5bedc0babf82a9a07d325494be7f5c3cfde56f43166b65f +│ └── f9d6350d0c44c0e7165a522155f53181ce8c163a6b8ead1f6baea22d1a8d8a78 +├── index.json +└── oci-layout + +3 directories, 10 files +``` + +As you can see, there are 3 missing files at `sample-app/blobs/sha256` folder, the missing 3 blobs are the blobs from the +`run-image` that were not downloaded but if you check your config file you'll notice you have the same number of layers as +when you export the full image. ## Implementation notes From e59664810dc645650b4ebf28f049d0b71c8f185d Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Fri, 31 Mar 2023 17:47:55 -0500 Subject: [PATCH 6/9] Apply suggestions from code review Co-authored-by: Joe Kimmel <86852107+joe-kimmel-vmw@users.noreply.github.com> Co-authored-by: Aidan Delaney Signed-off-by: Juan Bustamante --- .../app-developer-guide/export-to-oci-layout.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/content/docs/app-developer-guide/export-to-oci-layout.md b/content/docs/app-developer-guide/export-to-oci-layout.md index 4edf6bf32..5f5d4223e 100644 --- a/content/docs/app-developer-guide/export-to-oci-layout.md +++ b/content/docs/app-developer-guide/export-to-oci-layout.md @@ -6,7 +6,7 @@ summary="Learn how to export your application image to disk in OCI layout format
The OCI Image Layout is the directory structure for OCI content-addressable blobs and location-addressable references. -
See the specification
+
See the specification.
Exporting to OCI layout format is an **experimental** feature available on pack since version X.Y.Z @@ -25,7 +25,7 @@ Enable the experimental features on pack pack config experimental true ``` -You can confirm everything is fine, checking the `config.toml` file in your `PACK_HOME` installation folder, for example: +You can confirm everything is fine, checking the `config.toml` file in your `PACK_HOME` installation folder. For example: ```bash cat ~/.pack/config.toml @@ -37,7 +37,7 @@ The configuration shows the experimental mode was **enabled** and a local direct ### 2. Build the app -If you haven't already, please follow the steps to [build an app](/docs/app-developer-guide/build-an-app), once you managed to build a sample application you can try exporting the sample application to disk in OCI layout format. +Please first follow the steps to [build an app](/docs/app-developer-guide/build-an-app), once you have successfully built an application you can export the sample application to disk in OCI layout format. The OCI layout feature must be enabled using the convention `oci:` in the `` parameter when invoking `pack build`. @@ -53,7 +53,7 @@ It will save the image in a folder `./sample-app` created in your current direct **Congratulations!** -You can verify your application image was saved on disk in a folder called `./sample-app` in your current directory in OCI layout format, for example: +You can verify your application image was saved on disk in a folder called `./sample-app` in your current directory in OCI layout format. For example: ```bash tree sample-app @@ -77,7 +77,7 @@ sample-app 3 directories, 13 files ``` -If you are interested on keep playing with the image in OCI layout format, one tool you can take a look is [umoci](https://umo.ci/) it can help you to create a +If you want to keep playing with the image in OCI layout format, one tool you can take a look at is [umoci](https://umo.ci/). It can help you to create a [runtime bundler](https://github.com/opencontainers/runtime-spec) that can be executed with another tool like [runc](https://github.com/opencontainers/runc) --- @@ -87,7 +87,7 @@ If you are interested on keep playing with the image in OCI layout format, one ### Skip saving your run-image layers on disk Before using this option we suggest to remove your local layout directory (the one configured in your pack config.toml with the key `layout-repo-dir`) and -your application image folder (if you are planning to use the same folder), the reason for this is pack doesn't remove the blobs saved in the `layout-repo-dir` if you use the `--sparse` flag +your application image folder (if you are planning to use the same folder). The reason for this is pack doesn't remove the blobs saved in the `layout-repo-dir` if you use the `--sparse` flag If you don't need your `run-image` layers on disk, you can skip them using `--sparse` flag in your `pack build` command invocation. @@ -117,7 +117,7 @@ sample-app 3 directories, 10 files ``` -As you can see, there are 3 missing files at `sample-app/blobs/sha256` folder, the missing 3 blobs are the blobs from the +As you can see, there are 3 missing files at `sample-app/blobs/sha256` folder. The missing 3 blobs are the blobs from the `run-image` that were not downloaded but if you check your config file you'll notice you have the same number of layers as when you export the full image. From 8292b0ebba53e9cae80393f873b82487c8e3e0ed Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Mon, 14 Aug 2023 15:13:42 -0500 Subject: [PATCH 7/9] Update content/docs/app-developer-guide/export-to-oci-layout.md Co-authored-by: Aidan Delaney Signed-off-by: Juan Bustamante --- content/docs/app-developer-guide/export-to-oci-layout.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/app-developer-guide/export-to-oci-layout.md b/content/docs/app-developer-guide/export-to-oci-layout.md index 5f5d4223e..5cbe5984c 100644 --- a/content/docs/app-developer-guide/export-to-oci-layout.md +++ b/content/docs/app-developer-guide/export-to-oci-layout.md @@ -13,7 +13,7 @@ Exporting to OCI layout format is an **experimental** feature available on pack ### 1. Enable experimental feature -Verify your pack version is equal or greater than X.Y.Z +Verify your pack version is equal or greater than 0.29.0 ```bash pack version From 22e144f210739d0c4ac92bc4d9a8ff771728e195 Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Wed, 16 Aug 2023 09:23:14 -0500 Subject: [PATCH 8/9] updating against main Signed-off-by: Juan Bustamante --- .../create-buildpack/setup-local-environment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/buildpack-author-guide/create-buildpack/setup-local-environment.md b/content/docs/buildpack-author-guide/create-buildpack/setup-local-environment.md index 636c88afc..391bc9a6a 100644 --- a/content/docs/buildpack-author-guide/create-buildpack/setup-local-environment.md +++ b/content/docs/buildpack-author-guide/create-buildpack/setup-local-environment.md @@ -90,4 +90,4 @@ Server: Docker Engine - Community --- Next Step - \ No newline at end of file + From db5efd4323b7af51a1c9afbac8caae4fc71242ec Mon Sep 17 00:00:00 2001 From: Juan Bustamante Date: Thu, 17 Aug 2023 08:23:04 -0500 Subject: [PATCH 9/9] updating some feedback Signed-off-by: Juan Bustamante --- content/docs/features/experimental/_index.md | 6 ++++++ .../experimental}/export-to-oci-layout.md | 9 +++++---- 2 files changed, 11 insertions(+), 4 deletions(-) create mode 100644 content/docs/features/experimental/_index.md rename content/docs/{app-developer-guide => features/experimental}/export-to-oci-layout.md (91%) diff --git a/content/docs/features/experimental/_index.md b/content/docs/features/experimental/_index.md new file mode 100644 index 000000000..d86c4e808 --- /dev/null +++ b/content/docs/features/experimental/_index.md @@ -0,0 +1,6 @@ ++++ +title="Experimental" +weight=1 +include_summaries=true +expand=false ++++ diff --git a/content/docs/app-developer-guide/export-to-oci-layout.md b/content/docs/features/experimental/export-to-oci-layout.md similarity index 91% rename from content/docs/app-developer-guide/export-to-oci-layout.md rename to content/docs/features/experimental/export-to-oci-layout.md index 5cbe5984c..53c5254a3 100644 --- a/content/docs/app-developer-guide/export-to-oci-layout.md +++ b/content/docs/features/experimental/export-to-oci-layout.md @@ -9,11 +9,11 @@ summary="Learn how to export your application image to disk in OCI layout format
See the specification.
-Exporting to OCI layout format is an **experimental** feature available on pack since version X.Y.Z +Exporting to OCI layout format is an **experimental** feature available on pack since version 0.30.0 ### 1. Enable experimental feature -Verify your pack version is equal or greater than 0.29.0 +Verify your pack version is equal or greater than 0.30.0 ```bash pack version @@ -33,7 +33,8 @@ experimental = true layout-repo-dir = "<$HOME>/.pack/layout-repo" ``` -The configuration shows the experimental mode was **enabled** and a local directory to save images on disk was configured to path `<$HOME>/.pack/layout-repo`. +The configuration shows the experimental mode was **enabled** and a local directory to save images on disk was configured to path `<$HOME>/.pack/layout-repo`. `layout-repo-dir` is being used as a [local repository](https://github.com/buildpacks/rfcs/blob/main/text/0119-export-to-oci.md#how-it-works) +to save images requires by `pack build` command in OCI layout format. ### 2. Build the app @@ -143,7 +144,7 @@ But [crane](https://github.com/google/go-containerregistry/tree/main/cmd/crane) ```bash crane pull --format=oci ``` -It will give you `application/vnd.docker.distribution.manifest.list.v2+json`, which will fail +It will give you `application/vnd.docker.distribution.manifest.list.v2+json`, which will fail because of the [state of our current implementation](https://github.com/buildpacks/rfcs/pull/203#discussion_r1092449172), we will improve this behavior in future versions.