Experiments submitted from Katib Web UI fail with NotImplementedError

[yashjakhotiya]

/kind bug

What steps did you take and what happened:
Launched a hyperparameter tuning experiment from the Katib Web UI. All experiments failed. Google Cloud error reporting showed this -

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/grpc/_server.py", line 434, in _call_behavior
    response_or_iterator = behavior(argument, context)
  File "/usr/src/app/github.com/kubeflow/katib/pkg/apis/manager/v1alpha3/python/api_pb2_grpc.py", line 135, in ValidateAlgorithmSettings
    raise NotImplementedError('Method not implemented!')
NotImplementedError: Method not implemented!

Running the same experiment via Python SDK completes successfully.

What did you expect to happen:
The experiments to run without errors.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
The template used was the default trial template but with image changed.

Environment:

• Kubeflow version (kfctl version): kfctl v1.0.2-0-ga476281

• Minikube version (minikube version):

• Kubernetes version: (use kubectl version): Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.40", GitCommit:"3931253cc6ee52d73719f6b7417313f08799e8a6", GitTreeState:"clean", BuildDate:"2020-05-18T20:15:42Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release): Ubuntu 18.04

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/katib	0.69

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/katib	0.69

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[yashjakhotiya]

/assign @johnugeorge

[andreyvelich]

@yashjakhotiya Thank you for reporting this.
This error should be fine, it just said that we don't have validation method in Suggestion.
Even with this error, Experiment should be succeeded.

How did you submit Experiment from the UI?
Can you show yaml file, that you used?

[andreyvelich]

@yashjakhotiya If you are still facing this problem feel free to re-open the issue.

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/front-end	0.63

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/front-end	0.63

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.