Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Code and jobs to cleanup the the new auto-deployed blueprints. #655

Merged
merged 1 commit into from
May 8, 2020

Conversation

jlewi
Copy link
Contributor

@jlewi jlewi commented May 7, 2020

  • We want to auto-deploy the GCP blueprint (Setup autodeploy for GCP blueprints GoogleCloudPlatform/kubeflow-distribution#5)

  • We need to add logic and K8s resources to cleanup the blueprints so
    we don't run out of GCP quota.

  • Create cleanup_blueprints.py to cleanup auto_deployed blueprints.

    • Don't put this code in cleanup_ci.py because we want to be able to
      use fire and possibly python3 (not sure code in cleanup_ci is python3
      compatible)
  • Create a CLI create_context.py to create K8s config contexts. This will
    be used to get credentials to talk to the cleanup cluster when running
    on K8s.

  • Create a Tekton task to run the cleanup script. This is intended
    as a replacement for our existing K8s job (Use Tekton (instead of K8s jobs) to cleanup the Kubeflow CI project #654). There's a couple reasons
    to start using Tekton

    i) We are already using Tekton as part of AutoDeploy infrastructure.
    ii) We can leverage Tekton to handle git checkouts.
    iii) Tekton makes it easy to additional steps to do things like
    create the context.

    • This is a partial solution. This PR contains a Tekton pipeline
      that is only running cleanup for the blueprints.

      • To do all cleanup using Tekton we just need to a step or Task to
        run the existing cleanup-ci script. The only issue I forsee
        is that the Tekton pipeline runs in the kf-ci-v1 cluster and
        will need to be granted access to the kubeflow-testing cluster
        so we can cleanup Argo workflows in that cluster.
  • To run the Tekton pipeline regulary we create a cronjob that runs kubectl
    apply.

  • cnrm_clients.py is a quick hack to create a wrapper to make it
    easier to work with CNRM custom resources.

* We want to auto-deploy the GCP blueprint (GoogleCloudPlatform/kubeflow-distribution#5)

* We need to add logic and K8s resources to cleanup the blueprints so
  we don't run out of GCP quota.

* Create cleanup_blueprints.py to cleanup auto_deployed blueprints.
  * Don't put this code in cleanup_ci.py because we want to be able to
    use fire and possibly python3 (not sure code in cleanup_ci is python3
    compatible)

* Create a CLI create_context.py to create K8s config contexts. This will
  be used to get credentials to talk to the cleanup cluster when running
  on K8s.

* Create a Tekton task to run the cleanup script. This is intended
  as a replacement for our existing K8s job (kubeflow#654). There's a couple reasons
  to start using Tekton

  i) We are already using Tekton as part of AutoDeploy infrastructure.
  ii) We can leverage Tekton to handle git checkouts.
  iii) Tekton makes it easy to additional steps to do things like
       create the context.

  * This is a partial solution. This PR contains a Tekton pipeline
    that is only running cleanup for the blueprints.

    * To do all cleanup using Tekton we just need to a step or Task to
      run the existing cleanup-ci script. The only issue I forsee
      is that the Tekton pipeline runs in the kf-ci-v1 cluster and
      will need to be granted access to the kubeflow-testing cluster
      so we can cleanup Argo workflows in that cluster.

* To run the Tekton pipeline regulary we create a cronjob that runs kubectl
  apply.

* cnrm_clients.py is a quick hack to create a wrapper to make it
  easier to work with CNRM custom resources.
@kubeflow-bot
Copy link

This change is Reviewable

@jlewi
Copy link
Contributor Author

jlewi commented May 7, 2020

/assign @Bobgy

@Bobgy
Copy link
Contributor

Bobgy commented May 7, 2020

/lgtm

@jlewi
Copy link
Contributor Author

jlewi commented May 8, 2020

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlewi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit d778258 into kubeflow:master May 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants