Skip to content

Latest commit

 

History

History
321 lines (260 loc) · 9.21 KB

remote-artifacts.md

File metadata and controls

321 lines (260 loc) · 9.21 KB
Raw

title: remote-artifacts authors:

  • @otaviof

reviewers:

  • @SaschaSchwarze0
  • @qu1queee
  • @adambkaplan
  • @zhangtbj
  • @coreydaley

approvers:

  • TBD creation-date: 2020-10-02 last-updated: 2020-11-09 status: provisional

Shipwright Remote Artifacts

Build Enhancement Proposals have been moved into the Shipwright Community repository. This document holds an obsolete Enhancement Proposal, please refer to the up-to-date SHIP for more information.

Summary

Remote artifacts are dependencies of the software building process, they represent binaries or other data stored outside of the version control system (Git). Hence, they are required when dealing with container images as well, and this enhancement proposal focuses on adding a remote artifacts support to Shipwright's Build API.

Motivation

Give Shipwright's operator broader build use-case support by enhancing its capabilities to include the concept of Artifacts. In other words, remote entities that will be available for the image build process.

End users will be able to include remote artifacts as a build runtime dependency, artifacts controlled by remote systems will be downloaded before the build process starts. Those artifacts may be a pre-compile binary, jar files, war files, etc.

Goals

  • Provide means to declare remote artifacts, dependencies that can be employed on builds;
  • Create the mechanism to download and prepare remote artifacts for builds;

Non-Goals

  • Automate the upload of local artifacts into the cluster;
  • Manage remote artifacts;
  • Amend container images created by the given BuildStrategy;
  • Walking a remote directory tree, similarly to a git checkout;

Proposal

The enhancement proposal is centered around the idea of declaring external dependencies, here called "artifacts", and being able to use them during the container image building process.

Build Artifacts

The remote artifacts will be directly expressed in the body of a Build resource, as the following example:

---
apiVersion: build.dev/v1alpha1
kind: Build
metadata:
  name: license-file
spec:
  source:
    # ...
  sources:
    - name: license-file
      type: http
      url: https://licenses.company.com/customer-id/license.tar
      http:
        path: $(workspace)/licenses/license.tar

We will add sources section, side by side with current source. The idea is to accommodate both constructions in v1alpha1, and save API breaking changes for upcoming v1beta1.

The new sources will contain the following attributes:

  • name: source name (required);
  • type: input source type, git or http (required);
  • sourceRef: use a external resource, the name in this field is the resource name (optional);
  • url: the URL of the repository or remote artifact (optional);
  • credentials: the credentials to interact with the artifact (optional);
  • http: settings for http, containing inner attribute .spec.http.path (optional);
  • git: settings for git, as exists today;

The resource UID/GID will be defined by the same user running the Tekton task, later on, we can extend the API to support arbitrary configuration.

Single vs. Multiple Git Repositories

For the initial implementation, we will only support one git repository that is specified in .spec.source. Remote artifacts will be specified n .spec.sources only.

We will address supporting multiple Git repositories in a future enhancement proposal. This will involve re-defining /workspace/source location and the $workspace placeholder.

Steps and Helper

Artifacts are requirements for the build process, thus downloading and preparing artifacts must happen before the build process starts. To achieve the objective, the operator will generate a new task step to download the artifacts.

The download may happen using existing open-source software, more specifically wget. The container image which contains this software will be specified via environment variable, and when not informed will use a default value.

Therefore, the buildrun controller needs to generate a TaskRun step, having a wget commands plus arguments, to download the specified artifacts.

The step generated by the operator will look like the following:

---
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: artifact-download
spec:
  steps:
    - name: artifact-download
      image: busybox:latest
      command:
        - /bin/bash
      args:
        - -c
        - >
          wget --output="/workspace/source/classpath/main.jar" https://nexus.company.com/app/main.jar && \
          chown 1000:1001 /workspace/source/classpath/main.jar && \
          chmod 0644 /workspace/source/classpath/main.jar

Example Use-Case

Using Shipwright's proposed logos as example, let's assume we are building a TypeScript application which will use the project logo, and we would like to create two different builds, one with the default project logo, and another with the alternative.

By using remote artifacts, we can keep the separation of project source code and assets, and we can describe those resources as:

---
apiVersion: build.dev/v1alpha1
kind: BuildSource
metadata:
  name: ship-logo
spec:
  sources:
    - name: project-logo
      type: http
      url: https://user-images.githubusercontent.com/2587818/92114986-69bfb600-edfa-11ea-820e-96cdb1014f58.png
      http:
        path: $(workspace)/assets/images/shipwright-logo.png

And, the alternative logo:

---
apiVersion: build.dev/v1alpha1
kind: BuildSource
metadata:
  name: axes-logo
spec:
  sources:
    - name: project-logo
      type: http
      url: https://user-images.githubusercontent.com/2587818/92100668-c1ebbd80-ede4-11ea-9e8a-7379c3875ea0.png
      http:
        path: $(workspace)/assets/images/shipwright-logo.png

Then, we can create the Build resource, as per:

---
apiVersion: build.dev/v1alpha1
kind: Build
metadata:
  name: typescript-ex
spec:
  strategy:
    name: buildpacks-v3
    kind: ClusterBuildStrategy
  source:
    url: https://github.com/example/project.git
  sources:
    - name: source
      type: git
      url: https://github.com/otaviof/typescript-ex.git
    - name: project-logo
      sourceRef: ship-logo
  output:
    image: quay.io/otaviof/typescript-ex:latest

Now, we can create two BuildRun resources. The first only runs the build with original settings:

---
apiVersion: build.dev/v1alpha1
kind: BuildRun
metadata:
  name: typescript-ex
spec:
  buildRef:
    name: typescript-ex

And later, we can create yet another BuildRun, but this time use the alternative logo. Here we are overwriting the project-logo source name, with an alternative resource, i.e.:

---
apiVersion: build.dev/v1alpha1
kind: BuildRun
metadata:
  name: typescript-ex-alternative-logo
spec:
  buildRef:
    name: typescript-ex
  output:
    image: quay.io/otaviof/typescript-ex:alternative
  sources:
    - name: project-logo
      sourceRef: axes-logo

When the build processes are done, the following images will be available:

  • quay.io/otaviof/typescript-ex:latest
  • quay.io/otaviof/typescript-ex:alternative

A number of real world use-cases can be derived from this example, the BuildSource is the foundation.

Test Plan

  1. Deploy the Shipwright Build operator in a cluster;
  2. Create a BuildSource resource instance, point to a remote binary;
  3. Create Build and BuildRun resources, using BuildSource;
  4. Make sure the build process happens successfully, being able to use remote artifact;

Alternatives

Standalone CRD

Alternatively, we may define the artifacts as a standalone CRD, that is a BuildSource resource. The advantage of this design is being able to exchange, and reuse, artifacts on several builds. For instance, if two projects are sharing a common logo image, both Builds will refer to the same BuildSource.

Splitting up resources will imply on modifying Build and BuildRun resources, and include .spec.sourceRef. The new attribute will make possible linking the build with a BuildSource resource.

The following snippet shows how an Artifact, i.e.: BuildSource, will be represented.

---
apiVersion: build.dev/v1alpha1
kind: BuildSource
metadata:
  name: license-file
spec:
  sources:
    - name: license.tar
      type: http
      url: https://licenses.company.com/customer-id/license.tar
      http:
        path: $(workspace)/licenses/license.tar

Usage

The usage of this feature is based on declaring a slice of .spec.sources and later on overwriting entries using BuildRun resource. For instance:

---
apiVersion: build.dev/v1alpha1
kind: Build
metadata:
  name: example
spec:
  source:
    # ...
  sources:
    - name: license-file

Could have its license-file overwritten in a BuildRun with:

---
apiVersion: build.dev/v1alpha1
kind: BuildRun
metadata:
  name: license-file
spec:
  buildRef:
    name: example
  sources:
    - name: license-file
      sourceRef: alternative-file