add katib studyjob launcher #754

hougangliu · 2019-01-30T14:54:02Z

This change is

hougangliu · 2019-01-31T02:10:30Z

/retest kubeflow-pipeline-e2e-test

hougangliu · 2019-01-31T02:19:19Z

/assign @richardsliu

IronPan · 2019-02-05T23:37:11Z

/assign @Ark-kun @gaoning777

vicaire · 2019-02-13T07:41:50Z

Hello @hougangliu,

Do you think it would make sense to implement something more generic that could launch and wait for any CRD?

On top of this generic functionality, some convenience higher level methods could be provided (as the Katib one that you are proposing here). WDYT?

hougangliu · 2019-02-13T07:55:13Z

Hello @hougangliu,

Do you think it would make sense to implement something more generic that could launch and wait for any CRD?

On top of this generic functionality, some convenience higher level methods could be provided (as the Katib one that you are proposing here). WDYT?

It makes sense

vicaire · 2019-02-14T04:07:14Z

/cc @hongye-sun

Adding Hongye to this thread as he is looking into some related problem.

hougangliu · 2019-02-19T07:44:43Z

/retest continuous-integration

hougangliu · 2019-02-26T06:39:12Z

@gaoning777 @Ark-kun @hongye-sun any comment?

hougangliu · 2019-02-27T23:32:56Z

/retest

hougangliu · 2019-02-28T01:16:42Z

/assign @IronPan

Ark-kun · 2019-02-28T02:25:44Z

components/kubeflow/katib-launcher/src/launch_study_job.py

+  if wait_response.get("status", {}).get("condition") == "Completed":
+    succ = True
+    trial = get_best_trial(wait_response["status"]["bestTrialId"])
+    with open('/output.txt', 'w') as f:


Please get this path from the command line.

done! thanks!

Ark-kun · 2019-02-28T02:26:59Z

components/kubeflow/katib-launcher/src/launch_study_job.py

+                            metricsnames, parameterconfigs, nasConfig, workertemplatepath, mcollectortemplatepath, suggestionspec):
+  """_generate_studyjob_yaml generates studyjob yaml file based on hp.template.yaml"""
+  with open(src_filename, 'r') as f:
+    content = yaml.load(f)


Maybe you can just fully build the spec dict here in code instead of loading it from file and replacing some values?

just follow how tfjob-launcher fills spec dict. I will submit another PR to fix the both two components. Thanks!

vicaire · 2019-02-28T21:37:18Z

.cloudbuild.yaml

@@ -97,8 +97,12 @@ steps:
  id: 'buildDeployer'
 - name: 'gcr.io/cloud-builders/docker'
  entrypoint: '/bin/bash'
-  args: ['-c', 'cd /workspace/components/kubeflow/launcher && ./build_image.sh -p $PROJECT_ID -t $COMMIT_SHA']


Could we not build the container automatically for now? It requires a complicated review process on our side.

Behavior here will keep unchanged. I just rename launcher (in fact, it only launches tfjob) to tf-launcher since we can add more launchers (saying pytorch job launcher, studyjob launcher and so on)

Thank you @hougangliu. Could we do the renaming in a separate PR? There are several dependencies on this tfjob component so I would like to separate the renaming into another PR in case we need to rollback.

vicaire · 2019-02-28T21:38:11Z

.cloudbuild.yaml

@@ -97,8 +97,12 @@ steps:
  id: 'buildDeployer'
 - name: 'gcr.io/cloud-builders/docker'
  entrypoint: '/bin/bash'
-  args: ['-c', 'cd /workspace/components/kubeflow/launcher && ./build_image.sh -p $PROJECT_ID -t $COMMIT_SHA']
+  args: ['-c', 'cd /workspace/components/kubeflow/tf-launcher && ./build_image.sh -p $PROJECT_ID -t $COMMIT_SHA']


Is the change from launcher to tf-launcher necessary? It seems unrelated to this PR.

I just rename launcher (in fact, it only launches tfjob) to tf-launcher since we can add more launchers (saying pytorch job launcher, studyjob launcher and so on)

vicaire · 2019-02-28T21:40:39Z

@hongye-sun, could you please have a look at this one? I think you could have some good feedback as you are working on many components.

vicaire · 2019-03-01T23:19:11Z

.cloudbuild.yaml

@@ -97,8 +97,12 @@ steps:
  id: 'buildDeployer'
 - name: 'gcr.io/cloud-builders/docker'
  entrypoint: '/bin/bash'
-  args: ['-c', 'cd /workspace/components/kubeflow/launcher && ./build_image.sh -p $PROJECT_ID -t $COMMIT_SHA']


Thank you @hougangliu. Could we do the renaming in a separate PR? There are several dependencies on this tfjob component so I would like to separate the renaming into another PR in case we need to rollback.

vicaire · 2019-03-01T23:22:10Z

.cloudbuild.yaml

  id: 'buildLauncher'
+- name: 'gcr.io/cloud-builders/docker'
+  entrypoint: '/bin/bash'


Hi @hougangliu,

This entry is building the new container automatically. Could you please remove this entry? It requires a complicated review process on our side before we can release new containers automatically.

thanks, Done

undo tf-launcher

vicaire · 2019-03-04T21:49:10Z

/lgtm
/approve

k8s-ci-robot · 2019-03-04T21:49:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vicaire

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~components/OWNERS~~ [vicaire]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* add katib studyjob launcher * delete tmp file * fix link error to tf-laucher * import studyjob client from katib project * specify output file with a parameter undo tf-launcher

Some of the recent PRs. kubeflow#750 kubeflow#709 kubeflow#706 kubeflow#633 kubeflow#636 kubeflow#638 kubeflow#646

k8s-ci-robot added the size/L label Jan 30, 2019

k8s-ci-robot requested review from gaoning777, hongye-sun and Ark-kun January 30, 2019 14:54

k8s-ci-robot added size/XL and removed size/L labels Jan 30, 2019

hougangliu force-pushed the launch-study branch from f361365 to d682625 Compare January 31, 2019 02:18

k8s-ci-robot assigned richardsliu Jan 31, 2019

k8s-ci-robot assigned Ark-kun and gaoning777 Feb 5, 2019

hougangliu force-pushed the launch-study branch from d682625 to d1daf0b Compare February 8, 2019 00:47

hougangliu force-pushed the launch-study branch from d1daf0b to cf28873 Compare February 13, 2019 07:50

hougangliu force-pushed the launch-study branch from cf28873 to d4e49fc Compare February 19, 2019 06:12

k8s-ci-robot added size/L and removed size/XL labels Feb 19, 2019

hougangliu force-pushed the launch-study branch from d4e49fc to a9108ed Compare February 21, 2019 02:52

hougangliu force-pushed the launch-study branch from a9108ed to 46fee76 Compare February 27, 2019 05:11

Ark-kun reviewed Feb 28, 2019

View reviewed changes

vicaire reviewed Feb 28, 2019

View reviewed changes

vicaire assigned hongye-sun Feb 28, 2019

vicaire reviewed Mar 1, 2019

View reviewed changes

hougangliu added 4 commits March 4, 2019 07:22

add katib studyjob launcher

12f0309

delete tmp file

4c1cef2

fix link error to tf-laucher

e25f24d

import studyjob client from katib project

79b698a

hougangliu force-pushed the launch-study branch 2 times, most recently from 6bfd44b to 3eb4b1a Compare March 3, 2019 23:31

k8s-ci-robot added size/XXL and removed size/L labels Mar 3, 2019

specify output file with a parameter

4cad518

undo tf-launcher

hougangliu force-pushed the launch-study branch from 3eb4b1a to 4cad518 Compare March 3, 2019 23:37

k8s-ci-robot added size/L and removed size/XXL labels Mar 3, 2019

k8s-ci-robot assigned vicaire Mar 4, 2019

k8s-ci-robot added the lgtm label Mar 4, 2019

k8s-ci-robot added the approved label Mar 4, 2019

k8s-ci-robot merged commit 86c4d3b into kubeflow:master Mar 4, 2019

hougangliu mentioned this pull request Mar 8, 2019

Pipeline and Katib Integration kubeflow/katib#331

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add katib studyjob launcher #754

add katib studyjob launcher #754

hougangliu commented Jan 30, 2019 •

edited by jlewi

Loading

hougangliu commented Jan 31, 2019

hougangliu commented Jan 31, 2019

IronPan commented Feb 5, 2019

vicaire commented Feb 13, 2019

hougangliu commented Feb 13, 2019

vicaire commented Feb 14, 2019

hougangliu commented Feb 19, 2019

hougangliu commented Feb 26, 2019

hougangliu commented Feb 27, 2019

hougangliu commented Feb 28, 2019

Ark-kun Feb 28, 2019

hougangliu Feb 28, 2019

Ark-kun Feb 28, 2019

hougangliu Feb 28, 2019

vicaire Feb 28, 2019

hougangliu Feb 28, 2019 •

edited

Loading

vicaire Mar 1, 2019

vicaire Feb 28, 2019

hougangliu Feb 28, 2019

vicaire commented Feb 28, 2019

vicaire Mar 1, 2019

vicaire Mar 1, 2019

hougangliu Mar 3, 2019

vicaire commented Mar 4, 2019

k8s-ci-robot commented Mar 4, 2019

add katib studyjob launcher #754

add katib studyjob launcher #754

Conversation

hougangliu commented Jan 30, 2019 • edited by jlewi Loading

hougangliu commented Jan 31, 2019

hougangliu commented Jan 31, 2019

IronPan commented Feb 5, 2019

vicaire commented Feb 13, 2019

hougangliu commented Feb 13, 2019

vicaire commented Feb 14, 2019

hougangliu commented Feb 19, 2019

hougangliu commented Feb 26, 2019

hougangliu commented Feb 27, 2019

hougangliu commented Feb 28, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hougangliu Feb 28, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vicaire commented Feb 28, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vicaire commented Mar 4, 2019

k8s-ci-robot commented Mar 4, 2019

hougangliu commented Jan 30, 2019 •

edited by jlewi

Loading

hougangliu Feb 28, 2019 •

edited

Loading