Make the Tekton CD pipeline for profile controller run on KF release …

…cluster

* Get rid of the PVC used to pass the image digest file between the build
  and update manifests step

  * Creating a PVC just creates operational complexity

* We combine the build and update manifests step into one task. We can
  then use /workspace (a pod volume) to pass data like the image digest
  file between the steps

* Update pipelineRun to work with version 0.9 of Tekton
  * Field serviceAccount has been renamed serviceAccountName

  * TaskRun no longer supports outputImageDir so we remove it; we will
    have to use Tekton to pass the image digest file

* Remove the namespace.yaml and secrets.yaml from the kustomize package

  * The secrets should be created out of band and not checked in
  * So the behavior should be to deploy the kustomize package in a namespace
    that already exists with the appropriate secrets

  * Checking in secrets is confusing

    * If we check in dummy secrets then users will get confused about
      whether the secrets are valid or not

    * Furthermore, the file secrets.yaml is an invitation to end up checking
      the secrets into source control.

* Configure some values to use gcr.io/kubeflow-images-public

* Disable ISTIO sidecar in the pipelines

* For kaniko we don't need the secret to be named a certain way we just
  need to set GOOGLE_APPLICATION_CREDENTIALS to point to the correct value

* We change kaniko to use the user-gcp-sa secret that Kubeflow creates

* We shouldn't need an image pull secret since kubeflow-images-public is public
  * GOOGLE_APPLICATION_CREDENTIALS should be used for pushing images

* Change the name of the secret containing ssh credentials for kubeflow-bot
  to kubeflow-bot-github-ssh

* rebuild-manifests.sh should use /workspace to get the image digest
  rather than the PVC.

* Simplify rebuild-manifests.ssh

  * Tekton will mount the .ssh information in /tekton/home/.ssh
    so we just need to create a symbolic link to /root/.ssh

  * The image digest file should be fetched from /workspace and not some PVC.

  * Set GITHUB_TOKEN  environment variable using secrets so that we don't
    need to use kubectl get to fetch it

  * We need to make the clone of kubeflow/manifests a non-shallow clone
    before we can push changes to the remote repo

Next steps:

* This PR only updated the profile controller

* We need to refactor how the PipelineRun's are laid out

  * I think we may want the PipelineRun's to be separate from the reused
    resurces like Task

* rebuil-manifests.sh should only regenerate tests for changed files

* The created PRs don't satisfy the Kubeflow CLA check.

Related to: kubeflow/testing#450

components/base/README.md

@@ -10,7 +10,11 @@
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
-## Kubeflow CI with tektoncd pipelines
+## Kubeflow CD with tektoncd pipelines
+
+This directory contains Tekton pipelines intended to rebuild Kubeflow docker images 
+and open PRs to update Kubeflow kustomize manifests to use the newly built images.
+
 
 ### Use Cases
 
@@ -19,7 +23,7 @@ The following use cases can be run on the following components (should be run fr
 - `kustomize build --reorder none `*jupyter-web-app*`/ci    | kubectl apply -f -`
 - `kustomize build --reorder none `*notebook-controller*`/ci | kubectl apply -f -`
 - `kustomize build --reorder none `*profile-controller*`/ci | kubectl apply -f -
- 
+
 This uses TektonCD [pipelinerun](https://github.com/tektoncd/pipeline/blob/master/docs/pipelineruns.md) to enable the following use case:
 
 1. A PR is merged into kubeflow/kubeflow updating the component
@@ -59,11 +63,10 @@ In this use case the following instance is created:
    │       └── manifests+revision 
    └── pipeline
        └── tasks
-           ├── build-push
-           └── update-manifests
+           ├── build-push      
 ```
 
-The PipelineRun includes a Pipeline that has 2 tasks and 3 PipelineResources of type image (component) and git (kubeflow, manifests). The Tasks reference these resources in their inputs or outputs. 
+The PipelineRun includes a Pipeline that has 1 tasks and 3 PipelineResources of type image (component) and git (kubeflow, manifests). The Tasks reference these resources in their inputs or outputs. 
 
 ### Parameterization 
 
@@ -72,8 +75,7 @@ The Pipeline uses parameterized Tasks.
 Reusing this pipeline only requires changing parameters in params.env in the target component
 
 The parameters are noted below, those with an asterix should change per component:
-Those parameters without an asterix allow different gcr.io locations, namespace and pvc_mount_path.
-This can be run locally (for example using a local cluster via `kind create cluster`)
+Those parameters without an asterix allow different gcr.io locations and namespace.
 
 ```
   container_image=gcr.io/kubeflow-ci/test-worker:latest
@@ -91,13 +93,41 @@ This can be run locally (for example using a local cluster via `kind create clus
   pvc_mount_path=/kubeflow
 ```
 
-### Secrets
+### Setting up a cluster to run the pipelines
+
+The kustomize manifests are currently written so as to run in a Kubeflow releasing cluster.
+
+The current release cluster is
+
+* **project**: **kf-releasing**
+* **cluster**: **kf-releasing-0-6-2**
+* **namespace**: **kf-releasing**
+
+This is a Kubeflow cluster (v0.6.2) and we rely on that to configure certain things like the secrets and service accounts.
+
+1. Follow [Tektons' instructions](https://github.com/tektoncd/pipeline/blob/master/docs/auth.md#ssh-authentication-git) for
+   creating a secret containing ssh credentials for use with GitHub
+
+   * We are currently using the secret named **kubeflow-bot-github-ssh**
+
+
+1. Ensure the GCP service account used with Kaniko has storage admin permissions for the project
+   where the images are pushed.
+
+   * most likely **gcr.io/kubeflow-images-public**
+
+1. Create a secret named **github-token** containing a github token to be used by the hub CLI to create PRs.
+
+### Run a pipeline
+
+1. Modify `base/params.env`
+
+   * set namespace to the namespace where it will run
+
+1. Run
+
+   ```
+   kustomize build --reorder none `*profile-controller*`/ci | kubectl apply -f -
+   ```
 
-The secrets file has been supplied with no tokens and should have tokens generated. 
-The file itself should not be checked in with valid tokens. 
-- gcp-credentials
-- kaniko-secret (same as gcp-credentials, use by kaniko)
-- github-ssh
-- github-token
 
-For the github-ssh and github-token secrets the kubeflow-bot github user and it's forked repo should be used.

components/base/kustomization.yaml

@@ -1,9 +1,6 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-- namespace.yaml
-- persistent-volume-claim.yaml
-- secrets.yaml
 - service-account.yaml
 - role-binding.yaml
 - pipeline-resource.yaml

components/base/params.env

@@ -1,6 +1,6 @@
-namespace=kubeflow-ci
-container_image=gcr.io/constant-cubist-173123/test-worker@sha256:08cc88cf7cac0742f52822716ec5da8137d82378a2b05dc11e7d813c04d4c572
-image_url=gcr.io/constant-cubist-173123
+namespace=kf-releasing
+container_image=gcr.io/kubeflow-releasing/test-worker@sha256:35138a42b57160a078e802b7d69aec3c3e79a3e2e55518af7798275ebcc84d25
+image_url=gcr.io/kubeflow-images-public
 kubeflow_repo_revision=master
 kubeflow_repo_url=git@github.com:kubeflow/kubeflow.git
 manifests_repo_revision=master

components/base/pipeline.yaml

@@ -21,20 +21,6 @@ spec:
       value: $(path_to_context)
     - name: path_to_docker_file
       value: $(path_to_docker_file)
-    resources:
-      inputs:
-      - name: kubeflow
-        resource: kubeflow
-      outputs:
-      - name: $(image_name)
-        resource: $(image_name)
-    taskRef:
-      name: build-push
-      kind: namespaced
-  - name: update-manifests
-    runAfter:
-    - build-push
-    params:
     - name: container_image
       value: "$(container_image)"
     - name: path_to_manifests_dir
@@ -45,10 +31,30 @@ spec:
         resource: kubeflow
       - name: manifests
         resource: manifests
+      outputs:
       - name: $(image_name)
         resource: $(image_name)
-        from:
-        - build-push
     taskRef:
-      name: update-manifests
+      name: build-push
       kind: namespaced
+  #- name: update-manifests
+  #  runAfter:
+  #  - build-push
+  #  params:
+  #  - name: container_image
+  #    value: "$(container_image)"
+  #  - name: path_to_manifests_dir
+  #    value: "$(path_to_manifests_dir)"
+  #  resources:
+  #    inputs:
+  #    - name: kubeflow
+  #      resource: kubeflow
+  #    - name: manifests
+  #      resource: manifests
+  #    - name: $(image_name)
+  #      resource: $(image_name)
+  #      from:
+  #      - build-push
+  #  taskRef:
+  #    name: update-manifests
+  #    kind: namespaced