Mechanism to determine if image/manifests is updated #545

jlewi · 2019-12-13T15:35:04Z

In order to enable continuous delivery of our docker images and our kustomize manifests (#450)
We need a way to keep track of the source commit from which an image was built so we can determine if it is outdated.

There are a couple ways we could do this

Image tags
- image tags are currently used by https://github.com/kubeflow/kubeflow/blob/2da148904248d8616b50a0d8287742f2e2bdc968/py/kubeflow/kubeflow/ci/update_jupyter_web_app.py#L140
Docker image labels
labels in the kustomize manifest
- e.g. add a label to the Application corresponding to the source commit

jlewi · 2019-12-13T15:38:22Z

It looks like kaniko can support multiple tags just by adding multiple --destination flags
https://github.com/GoogleContainerTools/kaniko/blob/master/README.md

* kubeflow/testing is the central repository where all reusable engprod code goes. As a result it makes sense for the reusable Tekton definitions and scripts related to continuous building and updating of Kubeflow applications to live here * For now, we will also centralize the definition of pipelines for all applications in this repository (as opposed to having them live in the application repositories). * This should make it easier to manage and automatically update all applications. * Per kubeflow#544 redo how we use kustomize and Tekton to parameterize the pipelines. * Individual runs of pipelines will rely completely on Tekton parameter substitution to create a run to build an image for a specific application at a specific commit. * The image URL and git commit of source code will be Tekton PipelineResources that are inlined in the PipelineRun * For any parameters we will use Tekton parameters and inline the values in the PipelineRun * PipelineRuns will just be created as YAML files and not as kustomize overlays * Right now we have a single kustomize package which defines the reusable elements which are Tekton Tasks and Pipeline resources * Right now we have a single Pipeline for all applications but in the future we might have application specific pipelines * rebuild_manifests.sh should use the image tag v0.x.y-${commit} rather than the digest. * This image tagging scheme will be the basis for determining whether the image is already up to date (kubeflow#545) * since we now specify the full image_url rather than using image name we need a parameter for the src_image that is used with the kustomize edit function.

* kubeflow/testing is the central repository where all reusable engprod code goes. As a result it makes sense for the reusable Tekton definitions and scripts related to continuous building and updating of Kubeflow applications to live here * For now, we will also centralize the definition of pipelines for all applications in this repository (as opposed to having them live in the application repositories). * This should make it easier to manage and automatically update all applications. * Per kubeflow#544 redo how we use kustomize and Tekton to parameterize the pipelines. * Individual runs of pipelines will rely completely on Tekton parameter substitution to create a run to build an image for a specific application at a specific commit. * The image URL and git commit of source code will be Tekton PipelineResources that are inlined in the PipelineRun * For any parameters we will use Tekton parameters and inline the values in the PipelineRun * PipelineRuns will just be created as YAML files and not as kustomize overlays * Right now we have a single kustomize package which defines the reusable elements which are Tekton Tasks and Pipeline resources * Right now we have a single Pipeline for all applications but in the future we might have application specific pipelines * rebuild_manifests.sh should use the image tag v0.x.y-${commit} rather than the digest. * This image tagging scheme will be the basis for determining whether the image is already up to date (kubeflow#545) * since we now specify the full image_url rather than using image name we need a parameter for the src_image that is used with the kustomize edit function. * Use a separate task for updating the manifests * Now that we are using image tags of the form "{TAG}-{COMMIT}" which is determined at pipeline construction time; we no longer need to pass the digest file between the build-push step and the update manifests task which makes it much easier to run them as separate task since we don't need a pod volume to share data. Related to kubeflow#450 - CD pipelines for Kubeflow.

* kubeflow/testing is the central repository where all reusable engprod code goes. As a result it makes sense for the reusable Tekton definitions and scripts related to continuous building and updating of Kubeflow applications to live here * For now, we will also centralize the definition of pipelines for all applications in this repository (as opposed to having them live in the application repositories). * This should make it easier to manage and automatically update all applications. * Per #544 redo how we use kustomize and Tekton to parameterize the pipelines. * Individual runs of pipelines will rely completely on Tekton parameter substitution to create a run to build an image for a specific application at a specific commit. * The image URL and git commit of source code will be Tekton PipelineResources that are inlined in the PipelineRun * For any parameters we will use Tekton parameters and inline the values in the PipelineRun * PipelineRuns will just be created as YAML files and not as kustomize overlays * Right now we have a single kustomize package which defines the reusable elements which are Tekton Tasks and Pipeline resources * Right now we have a single Pipeline for all applications but in the future we might have application specific pipelines * rebuild_manifests.sh should use the image tag v0.x.y-${commit} rather than the digest. * This image tagging scheme will be the basis for determining whether the image is already up to date (#545) * since we now specify the full image_url rather than using image name we need a parameter for the src_image that is used with the kustomize edit function. * Use a separate task for updating the manifests * Now that we are using image tags of the form "{TAG}-{COMMIT}" which is determined at pipeline construction time; we no longer need to pass the digest file between the build-push step and the update manifests task which makes it much easier to run them as separate task since we don't need a pod volume to share data. Related to #450 - CD pipelines for Kubeflow.

stale · 2020-03-12T15:41:30Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in one week if no further activity occurs. Thank you for your contributions.

jlewi · 2020-03-17T14:41:39Z

Closing this. Our apps-cd pipeline has figured out a solution.

jlewi added priority/p1 kind/feature area/engprod labels Dec 13, 2019

jlewi mentioned this issue Dec 20, 2019

Move over the KF cd pipelines into kubeflow/testing and refactor #551

Merged

stale bot added the lifecycle/stale label Mar 12, 2020

jlewi closed this as completed Mar 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mechanism to determine if image/manifests is updated #545

Mechanism to determine if image/manifests is updated #545

jlewi commented Dec 13, 2019

jlewi commented Dec 13, 2019

stale bot commented Mar 12, 2020

jlewi commented Mar 17, 2020

Mechanism to determine if image/manifests is updated #545

Mechanism to determine if image/manifests is updated #545

Comments

jlewi commented Dec 13, 2019

jlewi commented Dec 13, 2019

stale bot commented Mar 12, 2020

jlewi commented Mar 17, 2020