Example with collecting timestamp of the metrics #970

andreyvelich · 2019-12-12T14:12:36Z

Fixes: #944.
Hold until I change docker hub repository.
/hold

This change is

…on-memlimit

…mp-metric-example

hougangliu · 2019-12-13T02:15:59Z

examples/v1alpha3/metrics-with-timestamp-example.yaml

+              spec:
+                containers:
+                - name: {{.Trial}}
+                  image: docker.io/andreyvelichkevich/timestamp-metric


I had added you into kubeflowkatib, you can now push kubeflowkatib/timestamp-metric now
BTW, I think you can rename the image name to kubeflowkatib/mxnet-mnist-example to make all example has timestamp.

I suggest to rename folder and file name also to metrics-with-timestamp to mxnet-mnist-example. There is no need to highlight the termwith-timestamp

This is not a mxnet-mnist-example.
I took code of pytorch-mnist from here: https://github.com/kubeflow/katib/blob/master/examples/v1alpha3/file-metrics-collector/mnist.py.

Should I change example to Mxnet in that case? Mxnet example is that one: https://github.com/apache/incubator-mxnet/blob/master/example/image-classification/train_mnist.py.
Or we can name this example kubeflowkatib/pytorch-mnist-example?

I missed it. IIRC @richardsliu created this image to show mxnet support.

@johnugeorge So what can we do with this example?

@richardsliu Can you point out at the Dockerfile used?

vpavlin · 2019-12-17T10:29:58Z

examples/v1alpha3/metrics-with-timestamp/Dockerfile

@@ -0,0 +1,6 @@
+FROM pytorch/pytorch:1.0-cuda10.0-cudnn7-runtime
+
+WORKDIR /var


I don't believe /var is appropriate location for the script. If you check FSH (https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard) /var should be used for changing files.

I would suggest using /opt and ideally a subdir in there - e.g. /opt/mnist/

WDYT?

Good catch. What do you think about /app?

Any particular reason you do not want to go with existing directory structure in /? I don't think it is a good practice in linux systems to put code in "random" dirs in /, but feel free to convince me otherwise:)

Do you mean in /examples/v1alpha3/metrics-with-timestamp ?

Do you mean in /examples/v1alpha3/metrics-with-timestamp ?

Is this answer to @johnugeorge'S question?:-)

Any particular reason you do not want to go with existing directory structure in /? I don't think it is a good practice in linux systems to put code in "random" dirs in /, but feel free to convince me otherwise:)

To this question :)
Which WORKDIR you think will be better?

Ah, sorry:) as I suggested in my first comment - I would go for /opt/mnist/mnist.py or to make it extremely obvious what it is (although that is already captured in the image name, but you never know..) /opt/metrics-with-timestamp/mnist.py or something along those lines

@johnugeorge @hougangliu What do you think about WORKDIR for the example as /opt/metrics-with-timestamp/mnist.py ?

…mp-metric-example

andreyvelich · 2020-01-08T18:03:34Z

I changed example from pytorch to mxnet.
I think that would be more useful for the Katib user.
Also, I use kubeflowkatib docker repository now.

I named it mxnet-mnist-timestamp, but we can think about better name.

/cc @johnugeorge @vpavlin
/hold cancel

k8s-ci-robot · 2020-01-08T18:03:37Z

@andreyvelich: GitHub didn't allow me to request PR reviews from the following users: vpavlin.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

I changed example from pytorch to mxnet.
I think that would be more useful for the Katib user.
Also, I use kubeflowkatib docker repository now.
/cc @johnugeorge @vpavlin
/hold cancel

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

andreyvelich · 2020-01-10T15:52:25Z

I changed mxnet-mnist-timestamp image to mxnet-mnist as we discussed here: #1008 (comment).
I think it will be better to name it mxnet-mnist than mxnet-mnist-example.
In case that some users use old yaml files for their Katib experiments with mxnet-mnist-example image and, if we push new image to mxnet-mnist-example, it will brake current Katib experiments.

I renamed images in according yaml files, please take a look.

/cc @johnugeorge @hougangliu @gaocegege

…mp-metric-example

andreyvelich · 2020-01-11T00:16:13Z

I changed CI cluster num nodes to 8.
It decreased CI test time ~ 30%.
What do you think about this change @johnugeorge @hougangliu @gaocegege ?

johnugeorge · 2020-01-13T12:58:49Z

test/scripts/v1alpha3/create-cluster.sh

@@ -33,6 +33,9 @@ echo "Creating GPU cluster"
 gcloud --project ${PROJECT} beta container clusters create ${CLUSTER_NAME} \
    --zone ${ZONE} \
    --machine-type=n1-standard-8 \
+    --num-nodes=8 \


This should not be changed since overall resources are fixed

johnugeorge · 2020-01-14T05:53:57Z

/lgtm

johnugeorge · 2020-01-14T05:54:12Z

/approve

k8s-ci-robot · 2020-01-14T05:54:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnugeorge

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [johnugeorge]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

andreyvelich added 8 commits December 9, 2019 18:06

Increase Suggestion memLimit

bebf6c0

Create getSuggestionConfigData function

f1bd3ca

Change memLimit for nasrl

cafd115

Merge remote-tracking branch 'upstream/master' into increase-suggesti…

2e57536

…on-memlimit

Change resources format for katib-config

f08fd42

Merge remote-tracking branch 'upstream/master' into issue-944-timesta…

c8acc67

…mp-metric-example

Create example with recording metrics timestamp

30eaad5

Merge remote-tracking branch 'upstream/master' into issue-944-timesta…

f840692

…mp-metric-example

k8s-ci-robot added the do-not-merge/hold label Dec 12, 2019

k8s-ci-robot requested review from jinan-zhou and garganubhav December 12, 2019 14:12

k8s-ci-robot added the size/L label Dec 12, 2019

Add comment line

1aa3a8e

hougangliu reviewed Dec 13, 2019

View reviewed changes

vpavlin reviewed Dec 17, 2019

View reviewed changes

andreyvelich added 2 commits January 8, 2020 15:30

Merge remote-tracking branch 'upstream/master' into issue-944-timesta…

d52bd9d

…mp-metric-example

Change example from pytorch to mxnet

923902a

k8s-ci-robot added size/XL and removed size/L labels Jan 8, 2020

Delete find_mxnet file

9954e6b

k8s-ci-robot removed the do-not-merge/hold label Jan 8, 2020

k8s-ci-robot requested a review from johnugeorge January 8, 2020 18:03

andreyvelich mentioned this pull request Jan 9, 2020

Add NAS RL training container to kubeflowkatib repository #1008

Merged

Change mxnet-mnist-timestamp to mxnet-mnist

d3cbdda

k8s-ci-robot requested review from gaocegege and hougangliu January 10, 2020 15:52

Merge remote-tracking branch 'upstream/master' into issue-944-timesta…

68979df

…mp-metric-example

andreyvelich added 5 commits January 10, 2020 16:56

Reduce num epochs in grid

b3ca4ae

Enable autoscaling in CI cluster

a714749

Add max nodes

933fb3c

Add num nodes 6

6710326

Increase num nodes

ccbd475

johnugeorge reviewed Jan 13, 2020

View reviewed changes

andreyvelich added 2 commits January 13, 2020 14:42

Change num nodes to 6

0c651a5

Remove autoscaling

f25c13d

k8s-ci-robot assigned johnugeorge Jan 14, 2020

k8s-ci-robot added the lgtm label Jan 14, 2020

k8s-ci-robot added the approved label Jan 14, 2020

k8s-ci-robot merged commit 2a8e8b4 into kubeflow:master Jan 14, 2020

andreyvelich deleted the issue-944-timestamp-metric-example branch October 6, 2021 00:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Example with collecting timestamp of the metrics #970

Example with collecting timestamp of the metrics #970

andreyvelich commented Dec 12, 2019 •

edited by jlewi

Loading

hougangliu Dec 13, 2019

johnugeorge Dec 13, 2019

andreyvelich Dec 13, 2019

johnugeorge Dec 13, 2019

andreyvelich Dec 16, 2019

johnugeorge Dec 17, 2019

vpavlin Dec 17, 2019

andreyvelich Dec 17, 2019

vpavlin Dec 17, 2019

andreyvelich Dec 17, 2019

vpavlin Dec 17, 2019

andreyvelich Dec 17, 2019

vpavlin Dec 17, 2019

andreyvelich Dec 18, 2019 •

edited

Loading

andreyvelich commented Jan 8, 2020 •

edited

Loading

k8s-ci-robot commented Jan 8, 2020

andreyvelich commented Jan 10, 2020

andreyvelich commented Jan 11, 2020

johnugeorge Jan 13, 2020

johnugeorge commented Jan 14, 2020

johnugeorge commented Jan 14, 2020

k8s-ci-robot commented Jan 14, 2020

		@@ -0,0 +1,6 @@
		FROM pytorch/pytorch:1.0-cuda10.0-cudnn7-runtime

		WORKDIR /var

Example with collecting timestamp of the metrics #970

Example with collecting timestamp of the metrics #970

Conversation

andreyvelich commented Dec 12, 2019 • edited by jlewi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreyvelich Dec 18, 2019 • edited Loading

Choose a reason for hiding this comment

andreyvelich commented Jan 8, 2020 • edited Loading

k8s-ci-robot commented Jan 8, 2020

andreyvelich commented Jan 10, 2020

andreyvelich commented Jan 11, 2020

Choose a reason for hiding this comment

johnugeorge commented Jan 14, 2020

johnugeorge commented Jan 14, 2020

k8s-ci-robot commented Jan 14, 2020

andreyvelich commented Dec 12, 2019 •

edited by jlewi

Loading

andreyvelich Dec 18, 2019 •

edited

Loading

andreyvelich commented Jan 8, 2020 •

edited

Loading