feat(components): Add v1beta1 Katib launcher and samples #4798

andreyvelich · 2020-11-21T05:29:48Z

Description of your changes:
Fixes: kubeflow/katib#1397.
Related: kubeflow/katib#1369.

I added Katib v1beta1 launcher and 2 samples under samples/contrib/kubeflow-katib dir:

Run Pipeline from the Jupyter Notebook using default component.yaml.
Compile Pipeline from the Python script and directly send parameters to the Katib launcher.

I use Base64Pickle type to send Experiment specification. I assume we can specify this type.
In that case, user can directly define Katib Experiment Spec using kubeflow-katib SDK which helps to avoid errors with invalid parameters.
What do you think ?

I have already pushed the new launcher to the registry: docker.io/kubeflowkatib/kubeflow-pipelines-launcher and tested 2 examples.

/assign @Bobgy @Ark-kun
/cc @gaocegege @johnugeorge @jlewi @Tomcli @fvde

Checklist:

The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

Do you want this pull request (PR) cherry-picked into the current release branch?

Learn more about cherry-picking updates into the release branch.

review-notebook-app · 2020-11-21T05:29:52Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

k8s-ci-robot · 2020-11-21T05:29:53Z

@andreyvelich: GitHub didn't allow me to request PR reviews from the following users: fvde.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

Description of your changes:
Fixes: kubeflow/katib#1397.
Related: kubeflow/katib#1369.

I added Katib v1beta1 launcher and 2 samples under samples/contrib/kubeflow-katib dir:

Run Pipeline from the Jupyter Notebook.

Compile Pipeline from the Python script.

I use Base64Pickle type to send Experiment specification. I assume we can specify this type.
In that case, user can directly define Katib Experiment Spec using kubeflow-katib SDK which helps to avoid errors with invalid parameters.
What do you think ?

I have already pushed the new launcher to the registry: docker.io/kubeflowkatib/kubeflow-pipelines-launcher and tested 2 examples.

/assign @Bobgy @Ark-kun
/cc @gaocegege @johnugeorge @jlewi @Tomcli @fvde

Checklist:

The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

Do you want this pull request (PR) cherry-picked into the current release branch?

Learn more about cherry-picking updates into the release branch.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Ark-kun · 2020-11-21T10:26:42Z

components/kubeflow/katib-launcher/component.yaml

- {name: Delete Finished Experiment, type: Bool,     default: 'True',   description: 'Whether to delete the experiment after it is finished.'}
+- {name: Experiment Name,            type: String,       default: '',        description: 'Experiment name'}
+- {name: Experiment Namespace,       type: String,       default: anonymous, description: 'Experiment namespace'}
+- {name: Experiment Spec,            type: Base64Pickle, default: '',        description: 'Experiment specification to convert in base64 pickle format'}


Can you use JsonObject type instead of Base64Pickle? Pickle is unfriendly to non-python components.

@Ark-kun Is it possible to convert V1beta1ExperimentSpec Object to JsonObject without losing nested class objects ?
When I use JsonObject and convert V1beta1ExperimentSpec to the dict, I can't restore the object structure in the Launcher.
I have to manually convert dict to the V1beta1ExperimentSpec object structure.

AFAIK, Kubernetes is very YAML-oriented and should support JSON. Perhaps there is some problem with conversion to a dictionary. Can you please check the naming style. Maybe the JSON uses snake_case instead of PascalCase.

Yes, that was one of the problem with converting objects.
Currently, I am using Katib SDK with sanitize_for_serialization to serialise ExperimentSpec object to JSON and deserialize to de-serialise JSON to ExperimentSpec object and submit Experiment.

Ark-kun · 2020-11-21T10:27:01Z

components/kubeflow/katib-launcher/component.yaml

 outputs:
- {name: Best Parameter Set,        type: JSON,                         description: 'The parameter set of the best Experiment trial.'}
+- {name: Best Parameter Set,         type: JSON,                             description: 'The hyperparameter set of the best Experiment Trial'}


JsonObject maybe?

Ark-kun · 2020-11-21T10:31:14Z

For long-term it might be great to refactor Katib to expose a trial suggestion library. This way we can avoid persistent mutable global state which will inevitable lead to problems.
See the design and comment in

pipelines/components/google-cloud/Optimizer/_samples/multi_stage_optimization_atomic_pipeline.py

Line 1 in b0a87e7

# This pipeline demonstrates hyper-parameter optimization.

gaocegege

LGTM!

andreyvelich · 2020-11-24T02:02:48Z

For long-term it might be great to refactor Katib to expose a trial suggestion library. This way we can avoid persistent mutable global state which will inevitable lead to problems.
See the design and comment in

pipelines/components/google-cloud/Optimizer/_samples/multi_stage_optimization_atomic_pipeline.py

Line 1 in b0a87e7

# This pipeline demonstrates hyper-parameter optimization.

@Ark-kun Does it mean that we want to provide user not only the best HP but also other parameters ?
For example, intermediate metrics, suggested parameters, Trials results.

Currently, Experiment lives as long as user doesn't delete it manually. Which means, user can resume it, etc.

Bobgy · 2020-11-24T08:15:44Z

also
/cc @chensun

andreyvelich · 2020-11-25T03:43:38Z

@Ark-kun I refactored Katib component Launcher to be able to process JsonObject in ExperimentSpec, can you take a look please ?

Also cc @terrytangyuan for the MPIJob example.

chensun · 2020-11-25T09:09:13Z

samples/contrib/kubeflow-katib/mpi-job-horovod.py

+    # Katib launcher component.
+    # We convert Experiment spec to JSON dict.
+    # The Experiment is deleted after the Pipeline is finished.
+    op = dsl.ContainerOp(


We suggest not to construct a ContainerOp instances directly. In fact, you should see a warning if you use it in this way. Can you please use an alternative?

pipelines/sdk/python/kfp/dsl/_container_op.py

Lines 1022 to 1027 in 60cf921

"Please create reusable components instead of constructing ContainerOp instances directly."

" Reusable components are shareable, portable and have compatibility and support guarantees."

" Please see the documentation: https://www.kubeflow.org/docs/pipelines/sdk/component-development/#writing-your-component-definition-file"

" The components can be created manually (or, in case of python, using kfp.components.create_component_from_func or func_to_container_op)"

" and then loaded using kfp.components.load_component_from_file, load_component_from_uri or load_component_from_text: "

"https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.components.html#kfp.components.load_component_from_file",

Sure @chensun.
I will follow the same way as in the Notebook example.

terrytangyuan

Thanks! @andreyvelich The MPI example looks great.

andreyvelich · 2020-11-27T19:31:19Z

/hold
Hold until we merge this PR: kubeflow/katib#1396. Check comment: kubeflow/katib#1396 (comment).

I'll try to use built-in functions to serialise and de-serialise ExperimentSpec object instead of writing the custom converter.

andreyvelich · 2020-11-30T15:51:02Z

/hold cancel
The kubeflow-katib 0.10.1 SDK version was updated.
This PR should be ready. Please take a look @Ark-kun @Bobgy @chensun.

/cc @gaocegege @johnugeorge

thomas-mueller · 2020-12-04T12:17:55Z

components/kubeflow/katib-launcher/src/launch_experiment.py

+    experiment_name = experiment.metadata.name
+    experiment_namespace = experiment.metadata.namespace
+    while True:
+        try:


Suggested change

try:

current_status = None

try:

This could be one way to prevent that current_status is undefined in case an exception is thrown.

Make sense, thanks @thomas-mueller!

Ark-kun · 2020-12-08T03:42:30Z

@Ark-kun Does it mean that we want to provide user not only the best HP but also other parameters ?
For example, intermediate metrics, suggested parameters, Trials results.

My idea is to have a "pure function" version of Katib which does not depend on mutable external state (Experiment resource). So, it would be great to have a function suggest_trials(parameter_specs, metric_specs, measurements_for_old_trials) -> List[Trial].

Currently, Experiment lives as long as user doesn't delete it manually. Which means, user can resume it, etc.

There are inherent issues with having mutable global state, global variables, etc. In KFP this becomes even more important since there is caching. The components are expected to be pure and produce the same results giving same input arguments.

Ark-kun · 2020-12-08T03:46:48Z

components/kubeflow/katib-launcher/build_image.sh

@@ -1,5 +1,5 @@
 #!/bin/bash -e
-# Copyright 2019 Google LLC
+# Copyright 2020 kubeflow.org.


Perhaps,

# Copyright 2020 The Kubeflow Authors

or

# Copyright 2020 Kubeflow contributors

andreyvelich · 2020-12-08T23:30:37Z

@Ark-kun Does it mean that we want to provide user not only the best HP but also other parameters ?
For example, intermediate metrics, suggested parameters, Trials results.

My idea is to have a "pure function" version of Katib which does not depend on mutable external state (Experiment resource). So, it would be great to have a function suggest_trials(parameter_specs, metric_specs, measurements_for_old_trials) -> List[Trial].

Currently, Experiment lives as long as user doesn't delete it manually. Which means, user can resume it, etc.

There are inherent issues with having mutable global state, global variables, etc. In KFP this becomes even more important since there is caching. The components are expected to be pure and produce the same results giving same input arguments.

That's a good idea.
One problem that I can see, which controller takes care about Trial concurrency, max Trial List, early stopped Trials, etc ?

Currently, Trials CR and Suggestion CR can't work without Experiment CR. Experiment CR asks for the requestNumber of Trials and Suggestion produces corresponding Trials assignments.
Then, Experiment controller creates appropriate Trials and Trial controller waits for the successful Trial's worker object status.

WDYT @gaocegege @johnugeorge ?

andreyvelich · 2020-12-11T00:21:37Z

@Ark-kun @Bobgy @chensun If you don't have any other comments on this PR should we merge it to move forward ?

Bobgy · 2020-12-15T06:00:48Z

I'll leave it to @chensun and @Ark-kun to make the decision.

Ark-kun · 2020-12-15T06:06:23Z

Thank you for your contribution and your patience. There was one unresolved review comment from me, but we can fix it afterwards.
/lgtm
/approve

k8s-ci-robot · 2020-12-15T06:06:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, Ark-kun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~components/kubeflow/katib-launcher/OWNERS~~ [Ark-kun]
~~samples/contrib/OWNERS~~ [Ark-kun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

andreyvelich added 5 commits November 21, 2020 04:33

Add Katib v1beta1 launcher and 2 examples

ba9a8de

Generate Notebook output

1415a87

Modify README

74e08f4

Fix links

41de148

Fix comments in mpi-job example

0645521

k8s-ci-robot assigned Ark-kun and Bobgy Nov 21, 2020

k8s-ci-robot requested review from gaocegege and johnugeorge November 21, 2020 05:29

k8s-ci-robot requested review from jlewi and Tomcli November 21, 2020 05:29

k8s-ci-robot added the size/XXL label Nov 21, 2020

google-cla bot added the cla: yes label Nov 21, 2020

Ark-kun reviewed Nov 21, 2020

View reviewed changes

gaocegege reviewed Nov 23, 2020

View reviewed changes

k8s-ci-robot requested a review from chensun November 24, 2020 08:15

andreyvelich added 2 commits November 25, 2020 03:39

Change Experiment spec structure to JsonObject

d52c75c

Remove generated file

1bb4d0f

chensun reviewed Nov 25, 2020

View reviewed changes

terrytangyuan reviewed Nov 25, 2020

View reviewed changes

Use component.yaml in MPI example

7f101c5

k8s-ci-robot added the do-not-merge/hold label Nov 27, 2020

andreyvelich added 2 commits November 27, 2020 21:26

Use built-in Katib API client serialization

e8a679d

Update notebook example

b1705c5

k8s-ci-robot removed the do-not-merge/hold label Nov 30, 2020

k8s-ci-robot requested a review from gaocegege November 30, 2020 15:51

thomas-mueller reviewed Dec 4, 2020

View reviewed changes

Assign current status to None

eb99a53

Ark-kun reviewed Dec 8, 2020

View reviewed changes

Fix Copyright

e14b9ba

k8s-ci-robot added the lgtm label Dec 15, 2020

k8s-ci-robot added the approved label Dec 15, 2020

k8s-ci-robot merged commit 89e4210 into kubeflow:master Dec 15, 2020

andreyvelich mentioned this pull request Jan 11, 2021

feat(components): Change Katib component output to JsonObject #4974

Merged

2 tasks

chinhuang007 mentioned this pull request Mar 15, 2021

E2E-mnist sample not up-to-date with Katib launcher #5306

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(components): Add v1beta1 Katib launcher and samples #4798

feat(components): Add v1beta1 Katib launcher and samples #4798

andreyvelich commented Nov 21, 2020 •

edited

Loading

review-notebook-app bot commented Nov 21, 2020

k8s-ci-robot commented Nov 21, 2020

Ark-kun Nov 21, 2020

andreyvelich Nov 24, 2020 •

edited

Loading

Ark-kun Dec 8, 2020

andreyvelich Dec 8, 2020

Ark-kun Nov 21, 2020

Ark-kun commented Nov 21, 2020

gaocegege left a comment

andreyvelich commented Nov 24, 2020

Bobgy commented Nov 24, 2020

andreyvelich commented Nov 25, 2020

chensun Nov 25, 2020

andreyvelich Nov 25, 2020

andreyvelich Nov 25, 2020

terrytangyuan left a comment

andreyvelich commented Nov 27, 2020

andreyvelich commented Nov 30, 2020

thomas-mueller Dec 4, 2020

andreyvelich Dec 4, 2020

Ark-kun commented Dec 8, 2020

Ark-kun Dec 8, 2020

andreyvelich commented Dec 8, 2020

andreyvelich commented Dec 11, 2020

Bobgy commented Dec 15, 2020

Ark-kun commented Dec 15, 2020

k8s-ci-robot commented Dec 15, 2020

	"Please create reusable components instead of constructing ContainerOp instances directly."
	" Reusable components are shareable, portable and have compatibility and support guarantees."
	" Please see the documentation: https://www.kubeflow.org/docs/pipelines/sdk/component-development/#writing-your-component-definition-file"
	" The components can be created manually (or, in case of python, using kfp.components.create_component_from_func or func_to_container_op)"
	" and then loaded using kfp.components.load_component_from_file, load_component_from_uri or load_component_from_text: "
	"https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.components.html#kfp.components.load_component_from_file",

feat(components): Add v1beta1 Katib launcher and samples #4798

feat(components): Add v1beta1 Katib launcher and samples #4798

Conversation

andreyvelich commented Nov 21, 2020 • edited Loading

review-notebook-app bot commented Nov 21, 2020

k8s-ci-robot commented Nov 21, 2020

Choose a reason for hiding this comment

andreyvelich Nov 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ark-kun commented Nov 21, 2020

gaocegege left a comment

Choose a reason for hiding this comment

andreyvelich commented Nov 24, 2020

Bobgy commented Nov 24, 2020

andreyvelich commented Nov 25, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

terrytangyuan left a comment

Choose a reason for hiding this comment

andreyvelich commented Nov 27, 2020

andreyvelich commented Nov 30, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ark-kun commented Dec 8, 2020

Choose a reason for hiding this comment

andreyvelich commented Dec 8, 2020

andreyvelich commented Dec 11, 2020

Bobgy commented Dec 15, 2020

Ark-kun commented Dec 15, 2020

k8s-ci-robot commented Dec 15, 2020

andreyvelich commented Nov 21, 2020 •

edited

Loading

andreyvelich Nov 24, 2020 •

edited

Loading