Support leader election for katib-controller #1713

tenzen-y · 2021-10-13T06:45:50Z

What this PR does / why we need it:
Support leader election for katib-controller.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1705

Checklist:

Docs included if any changes are user facing

/assign
/assign @andreyvelich @gaocegege @johnugeorge

coveralls · 2021-10-13T06:53:09Z

Coverage decreased (-0.05%) to 73.523% when pulling 7798d48 on tenzen-y:issue-1705-support-leader-election-for-katib-controller into 0a38e87 on kubeflow:master.

gaocegege

LGTM

Thanks for your contribution! 🎉 👍

gaocegege · 2021-10-13T06:58:59Z

/hold

/cc @andreyvelich

cmd/katib-controller/v1beta1/main.go

google-cla · 2021-10-13T07:26:35Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

tenzen-y · 2021-10-13T07:27:50Z

@googlebot I consent.

google-cla · 2021-10-13T07:28:02Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

tenzen-y · 2021-10-13T07:29:41Z

Can you please comment with @googlebot I consent.? @gaocegege

google-cla · 2021-10-13T07:29:46Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

gaocegege · 2021-10-13T07:40:08Z

@googlebot I consent.

tenzen-y · 2021-10-13T08:16:14Z

/retest

tenzen-y · 2021-10-13T10:12:24Z

I fixed setup-katib in e2e.

Failed log

job.batch/katib-cert-generator created
persistentvolumeclaim/katib-mysql created
validatingwebhookconfiguration.admissionregistration.k8s.io/katib.kubeflow.org created
Pod Status 0/7
All Katib components are running.
Katib deployments
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
katib-controller   2/2     2            2           13s
katib-db-manager   1/1     1            1           13s
katib-mysql        0/1     1            0           13s
katib-ui           1/1     1            1           13s
pytorch-operator   1/1     1            1           15s
tf-job-operator    1/1     1            1           16s

Successful log

job.batch/katib-cert-generator created
persistentvolumeclaim/katib-mysql created
validatingwebhookconfiguration.admissionregistration.k8s.io/katib.kubeflow.org created
Pod Status 0/8
Pod Status 6/8
Pod Status 6/8
All Katib components are running.
Katib deployments
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
katib-controller   2/2     2            2           34s
katib-db-manager   1/1     1            1           34s
katib-mysql        0/1     1            0           34s
katib-ui           1/1     1            1           34s
pytorch-operator   1/1     1            1           36s
tf-job-operator    1/1     1            1           37s

tenzen-y · 2021-10-13T10:28:08Z

These errors are caused by the pull rate limit in the docker hub.

https://argo.kubeflow-testing.com/workflows/kubeflow-test-infra/kubeflow-katib-presubmit-e2e-v1beta1-1713-c81858e-8128-f156?tab=workflow&nodeId=kubeflow-katib-presubmit-e2e-v1beta1-1713-c81858e-8128-f156-486381905&sidePanel=logs%3Akubeflow-katib-presubmit-e2e-v1beta1-1713-c81858e-8128-f156-486381905%3Amain

TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authentica
ting and upgrading: https://www.docker.com/increase-rate-limit

tenzen-y · 2021-10-13T14:39:43Z

/retest

johnugeorge · 2021-10-22T09:34:35Z

/lgtm
/approve

google-oss-robot · 2021-10-22T09:34:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gaocegege, johnugeorge, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [gaocegege,johnugeorge]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

andreyvelich

Thank you for pushing this @tenzen-y!
I left few comments.
/hold for the review

test/e2e/v1beta1/scripts/setup-katib.sh

manifests/v1beta1/components/controller/controller.yaml

tenzen-y · 2021-10-27T18:25:15Z

I modified scripts for the e2e test and fixed directory structure for HA manifests.
Can you take a look changes? @andreyvelich

scripts/v1beta1/deploy.sh

manifests/v1beta1/installs/ha/katib-cert-manager/kustomization.yaml

manifests/v1beta1/installs/ha/overlays/rbac-ha.yaml

tenzen-y · 2021-10-28T16:25:05Z

@andreyvelich
I've modified it based on your suggestions.
Can you check the changes, please?

If it looks good to you, I would like to create PR in the website repository to add docs for katib-leader-election to katib-setup section.

andreyvelich

Thank you for the updates @tenzen-y!
Small comments from me.

manifests/v1beta1/installs/katib-leader-election/leader-election-rbac.yaml

…on-rbac.yaml Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

google-cla · 2021-10-28T22:22:20Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

tenzen-y · 2021-10-28T22:23:24Z

Can you comment with @googlebot I consent., please? @andreyvelich

google-cla · 2021-10-28T22:23:40Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

andreyvelich · 2021-10-28T22:27:31Z

@googlebot I consent.

…on-rbac.yaml Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

andreyvelich · 2021-10-28T22:48:51Z

Thank you for pushing this @tenzen-y!
Please submit a PR to update the Kubeflow website with this new install as you mentioned before: #1713 (comment).

/lgtm
/assign @gaocegege @johnugeorge

johnugeorge · 2021-11-02T19:03:38Z

/approve

google-oss-prow · 2021-11-02T19:03:57Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gaocegege, johnugeorge, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [gaocegege,johnugeorge]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

andreyvelich · 2021-11-02T19:13:10Z

I think we can merge it.
Thank you for your contribution @tenzen-y!
/hold cancel

google-oss-robot assigned andreyvelich, gaocegege, johnugeorge and tenzen-y Oct 13, 2021

google-oss-robot requested review from andreyvelich, johnugeorge and sperlingxx October 13, 2021 06:45

google-oss-robot added the size/M label Oct 13, 2021

gaocegege approved these changes Oct 13, 2021

View reviewed changes

google-oss-robot added the approved label Oct 13, 2021

google-oss-robot added the do-not-merge/hold label Oct 13, 2021

gaocegege reviewed Oct 13, 2021

View reviewed changes

cmd/katib-controller/v1beta1/main.go Outdated Show resolved Hide resolved

tenzen-y force-pushed the issue-1705-support-leader-election-for-katib-controller branch from 0a3296d to c81858e Compare October 13, 2021 09:55

google-oss-robot added the lgtm label Oct 22, 2021

andreyvelich reviewed Oct 25, 2021

View reviewed changes

test/e2e/v1beta1/scripts/setup-katib.sh Outdated Show resolved Hide resolved

manifests/v1beta1/components/controller/controller.yaml Outdated Show resolved Hide resolved

support leader election for katib-controller

93f0bd9

google-oss-robot added the size/L label Oct 27, 2021

fix e2e test

b72daa8

andreyvelich reviewed Oct 27, 2021

View reviewed changes

scripts/v1beta1/deploy.sh Outdated Show resolved Hide resolved

manifests/v1beta1/installs/ha/katib-cert-manager/kustomization.yaml Outdated Show resolved Hide resolved

manifests/v1beta1/installs/ha/overlays/rbac-ha.yaml Outdated Show resolved Hide resolved

google-oss-robot added size/M and removed size/L labels Oct 28, 2021

tenzen-y force-pushed the issue-1705-support-leader-election-for-katib-controller branch 2 times, most recently from a48b4f1 to bd18869 Compare October 28, 2021 16:24

review: turn off leader-election by default

c55bd42

tenzen-y force-pushed the issue-1705-support-leader-election-for-katib-controller branch from bd18869 to c55bd42 Compare October 28, 2021 16:40

andreyvelich reviewed Oct 28, 2021

View reviewed changes

tenzen-y and others added 3 commits October 29, 2021 07:19

Update manifests/v1beta1/installs/katib-leader-election/leader-electi…

778fe45

…on-rbac.yaml Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Update manifests/v1beta1/installs/katib-leader-election/leader-electi…

4f99fb4

…on-rbac.yaml Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Update manifests/v1beta1/installs/katib-leader-election/leader-electi…

ee8fd73

…on-rbac.yaml Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

tenzen-y force-pushed the issue-1705-support-leader-election-for-katib-controller branch from 3127f1f to ee8fd73 Compare October 28, 2021 22:22

Update manifests/v1beta1/installs/katib-leader-election/leader-electi…

7798d48

…on-rbac.yaml Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

google-oss-robot added the lgtm label Oct 28, 2021

tenzen-y mentioned this pull request Oct 29, 2021

Add docs to describe how to install HA katib-controller kubeflow/website#3049

Merged

google-oss-prow bot removed the do-not-merge/hold label Nov 2, 2021

google-oss-robot merged commit f802295 into kubeflow:master Nov 2, 2021

tenzen-y deleted the issue-1705-support-leader-election-for-katib-controller branch November 3, 2021 03:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support leader election for katib-controller #1713

Support leader election for katib-controller #1713

tenzen-y commented Oct 13, 2021 •

edited

Loading

coveralls commented Oct 13, 2021 •

edited

Loading

gaocegege left a comment

gaocegege commented Oct 13, 2021

google-cla bot commented Oct 13, 2021

tenzen-y commented Oct 13, 2021

google-cla bot commented Oct 13, 2021

tenzen-y commented Oct 13, 2021

google-cla bot commented Oct 13, 2021

gaocegege commented Oct 13, 2021

tenzen-y commented Oct 13, 2021

tenzen-y commented Oct 13, 2021

tenzen-y commented Oct 13, 2021

tenzen-y commented Oct 13, 2021

johnugeorge commented Oct 22, 2021

google-oss-robot commented Oct 22, 2021

andreyvelich left a comment

tenzen-y commented Oct 27, 2021

tenzen-y commented Oct 28, 2021 •

edited

Loading

andreyvelich left a comment

google-cla bot commented Oct 28, 2021

tenzen-y commented Oct 28, 2021

google-cla bot commented Oct 28, 2021

andreyvelich commented Oct 28, 2021

andreyvelich commented Oct 28, 2021

johnugeorge commented Nov 2, 2021

google-oss-prow bot commented Nov 2, 2021

andreyvelich commented Nov 2, 2021

Support leader election for katib-controller #1713

Support leader election for katib-controller #1713

Conversation

tenzen-y commented Oct 13, 2021 • edited Loading

coveralls commented Oct 13, 2021 • edited Loading

gaocegege left a comment

Choose a reason for hiding this comment

gaocegege commented Oct 13, 2021

google-cla bot commented Oct 13, 2021

tenzen-y commented Oct 13, 2021

google-cla bot commented Oct 13, 2021

tenzen-y commented Oct 13, 2021

google-cla bot commented Oct 13, 2021

gaocegege commented Oct 13, 2021

tenzen-y commented Oct 13, 2021

tenzen-y commented Oct 13, 2021

tenzen-y commented Oct 13, 2021

tenzen-y commented Oct 13, 2021

johnugeorge commented Oct 22, 2021

google-oss-robot commented Oct 22, 2021

andreyvelich left a comment

Choose a reason for hiding this comment

tenzen-y commented Oct 27, 2021

tenzen-y commented Oct 28, 2021 • edited Loading

andreyvelich left a comment

Choose a reason for hiding this comment

google-cla bot commented Oct 28, 2021

tenzen-y commented Oct 28, 2021

google-cla bot commented Oct 28, 2021

andreyvelich commented Oct 28, 2021

andreyvelich commented Oct 28, 2021

johnugeorge commented Nov 2, 2021

google-oss-prow bot commented Nov 2, 2021

andreyvelich commented Nov 2, 2021

tenzen-y commented Oct 13, 2021 •

edited

Loading

coveralls commented Oct 13, 2021 •

edited

Loading

tenzen-y commented Oct 28, 2021 •

edited

Loading