Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Test/sample] Fix the model version in kubeflow_tf_serving sample to avoid potential conflict. #2170

Merged

Conversation

numerology
Copy link

@numerology numerology commented Sep 19, 2019

When there is no new commit detected in a PR, PULL_PULL_SHA will be reused, causing conflict in the export model path.

This can be fixed by using timestamp as the version number. See #2047 for context.


This change is Reviewable

@gaoning777
Copy link
Contributor

Adding the randomization in the sample itself is not a good practice. It's supposed to be the sample test infra's work to pass in different model names due to cluster sharing among test retries.

Could we start with a placeholder implementation? since the model name as well as the input/output could potentially use this placeholder.

@gaoning777
Copy link
Contributor

However, to unblock the team, we can merge this one now.
Could you create an issue?
/lgtm

@numerology
Copy link
Author

Adding the randomization in the sample itself is not a good practice. It's supposed to be the sample test infra's work to pass in different model names due to cluster sharing among test retries.

Could we start with a placeholder implementation? since the model name as well as the input/output could potentially use this placeholder.

For the placeholder, I have two options in mind:

  1. short term: we can dynamically replace some parameters to inject the notebook in sample test infra;
  2. long term: switch to jsonnet for test config files.

Shall we begin with the first one to unblock things?

@gaoning777
Copy link
Contributor

That sounds great. We can start with two placeholders: TIMESTAMP and COMMITTAG.
e.g.
test_name: kubeflow_tf_serving
notebook_params:
output: gs://ml-pipeline-test/${COMMITTAG}
project: ml-pipeline-test
experiment_name: kubeflow_tf_serving-test
model_name: model-${TIMESTAMP}

becomes
test_name: kubeflow_tf_serving
notebook_params:
output: gs://ml-pipeline-test/a86ee2dfbb20c67964130ec132c304bb3eaa010e
project: ml-pipeline-test
experiment_name: kubeflow_tf_serving-test
model_name: model-1568914870

@numerology
Copy link
Author

That sounds great. We can start with two placeholders: TIMESTAMP and COMMITTAG.
e.g.
test_name: kubeflow_tf_serving
notebook_params:
output: gs://ml-pipeline-test/${COMMITTAG}
project: ml-pipeline-test
experiment_name: kubeflow_tf_serving-test
model_name: model-${TIMESTAMP}

becomes
test_name: kubeflow_tf_serving
notebook_params:
output: gs://ml-pipeline-test/a86ee2dfbb20c67964130ec132c304bb3eaa010e
project: ml-pipeline-test
experiment_name: kubeflow_tf_serving-test
model_name: model-1568914870

Sounds good, can you help me approve plz? Thanks.

@gaoning777
Copy link
Contributor

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gaoning777

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 6e95603 into kubeflow:master Sep 19, 2019
@numerology numerology deleted the fix-kf-tf-serving-sample-path-conflict branch September 19, 2019 18:03
@gaoning777
Copy link
Contributor

Please link the issue you created here to keep in track.

@numerology
Copy link
Author

Please link the issue you created here to keep in track.

Triaging issue #2171

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants