Support for grid search algorithm in Optuna Suggestion Service #2060

tenzen-y · 2022-12-13T08:16:10Z

Signed-off-by: Yuki Iwai yuki.iwai.tz@gmail.com

What this PR does / why we need it:
I added a mechanism to support grid search in Optuna Suggestion Service since we will remove Chocolate Suggestion Service as mentioned in #2058.

Also, I switched the default Suggestion Service for grid search to Optuna Suggestion Service in katib-config.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Part of #2058

Checklist:

Docs included if any changes are user facing

google-oss-prow · 2022-12-13T08:16:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tenzen-y · 2022-12-13T20:58:47Z

/hold for the review

/assign @andreyvelich @johnugeorge @anencore94

tenzen-y · 2022-12-13T21:19:09Z

test/unit/v1beta1/suggestion/test_optuna_service.py

-        assert code == grpc.StatusCode.INVALID_ARGUMENT
-
-        # [RANDOM] Invalid random_state
+    @pytest.mark.parametrize(


IMO, we would be good to do refactoring unit tests using @pytest.mark.parametrize in other suggestion services, same as this.

We can follow up with other PRs.

I like this feature of PyTest, thanks!

pkg/suggestion/v1beta1/optuna/service.py

pkg/suggestion/v1beta1/internal/search_space.py

anencore94 · 2022-12-17T15:08:14Z

pkg/suggestion/v1beta1/optuna/service.py

+            if max_trial_count > num_combinations:
+                return False, "Max Trial Count: {max_trial} > all possible search combinations: {combinations}".\
+                    format(max_trial=max_trial_count, combinations=num_combinations)


I'm not familiar with optuna, Is it ok to set max_trial_count < num_combinations ? since with the documentation GridSampler looks like always trying to search all possible combinations.

https://optuna.readthedocs.io/en/latest/reference/samplers/generated/optuna.samplers.GridSampler.html

Yes, that's right about optuna behavior. Although optuna suggestion-service creates trials using already used parameters after all combinations used.

katib/pkg/suggestion/v1beta1/optuna/base_service.py

Lines 58 to 65 in db72ce1

for _ in range(current_request_number):

optuna_trial = self.study.ask(fixed_distributions=self._get_optuna_search_space())

assignments = [Assignment(k, v) for k, v in optuna_trial.params.items()]

list_of_assignments.append(assignments)

assignments_key = self._get_assignments_key(assignments)

self.assignments_to_optuna_number[assignments_key].append(optuna_trial.number)

This does not mean the optuna suggestion service can remove duplicated suggestions in the above section since the suggestionclient faces the error in the following:

katib/pkg/controller.v1beta1/suggestion/suggestionclient/suggestionclient.go

Lines 123 to 128 in db72ce1

logger.Info("Getting suggestions", "endpoint", endpoint, "Number of current request parameters", currentRequestNum, "Number of response parameters", len(responseSuggestion.ParameterAssignments))

if len(responseSuggestion.ParameterAssignments) != currentRequestNum {

err := fmt.Errorf("The response contains unexpected trials")

logger.Error(err, "The response contains unexpected trials")

return err

}

Also, if users want to run experiments using all combinations, users can skip setting maxTrialCount in the Experiment.

For these reasons, it would be good to provide that validation.

@anencore94 Does that make sense?

Yeap I see now. Thanks for your kind reply :)

tenzen-y · 2022-12-22T17:09:16Z

PTAL
/assign @kubeflow/wg-automl-leads

andreyvelich

Thank you for this effort @tenzen-y!
We should also update the docs:

andreyvelich · 2022-12-23T12:29:06Z

pkg/suggestion/v1beta1/internal/search_space.py

+                combinations[parameter.name] = range(int(parameter.min), int(parameter.max)+1, int(parameter.step))
+            elif parameter.type == DOUBLE:
+                if parameter.step == "" or parameter.step is None:
+                    raise Exception("Param {} step is nil".format(parameter.name))


Should we also say that: "For discrete search space all parameters must include step" ?

Sounds good. It helps to debug parameters.

andreyvelich · 2022-12-23T12:35:11Z

test/unit/v1beta1/suggestion/test_optuna_service.py

-        assert code == grpc.StatusCode.INVALID_ARGUMENT
-
-        # [RANDOM] Invalid random_state
+    @pytest.mark.parametrize(


I like this feature of PyTest, thanks!

tenzen-y · 2022-12-23T15:10:36Z

Thank you for this effort @tenzen-y! We should also update the docs:

https://www.kubeflow.org/docs/components/katib/experiment/#grid-search

https://www.kubeflow.org/docs/components/katib/katib-config/#suggestion-settings

@andreyvelich Thanks for the review and pointing out sections we need to update the documentation!
I will add a PR to update the documentation.

tenzen-y · 2022-12-23T15:12:00Z

pkg/suggestion/v1beta1/internal/search_space.py

+                    raise Exception(
+                        "Param {} step is nil; For discrete search space, all parameters must include step".
+                        format(parameter.name)
+                    )


@andreyvelich I updated this error message.

tenzen-y · 2022-12-23T15:13:00Z

@andreyvelich I addressed your comments; please take another look.

andreyvelich · 2022-12-23T15:44:50Z

Thank you @tenzen-y! We are going to remove Chocolate references in the following PRs, right ?
/lgtm
/assign @johnugeorge @anencore94

tenzen-y · 2022-12-23T15:46:21Z

Thank you @tenzen-y! We are going to remove Chocolate references in the following PRs, right ? /lgtm /assign @johnugeorge @anencore94

Yes, I will create another PR to remove all codes for Chocolate after this PR is merged.

tenzen-y · 2022-12-23T18:07:05Z

Blocked by: #2070

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

tenzen-y · 2022-12-24T04:33:26Z

@johnugeorge @anencore94 I rebased to take #2070. PTAL.

johnugeorge · 2022-12-24T05:46:46Z

/lgtm

Thanks @tenzen-y

tenzen-y · 2022-12-24T06:16:34Z

Thanks for all reviews and comments.
/hold cancel

google-oss-prow bot added the do-not-merge/work-in-progress label Dec 13, 2022

google-oss-prow bot requested review from andreyvelich, anencore94 and johnugeorge December 13, 2022 08:16

google-oss-prow bot added approved size/S labels Dec 13, 2022

tenzen-y changed the title ~~[WIP] Support grid search in Optuna Suggestion Service~~ [WIP] Support for grid search algorithm in Optuna Suggestion Service Dec 13, 2022

tenzen-y force-pushed the support-for-grid-search-in-optuna branch from 04b7d02 to 31b2d82 Compare December 13, 2022 19:07

google-oss-prow bot added size/M size/L and removed size/S size/M labels Dec 13, 2022

tenzen-y force-pushed the support-for-grid-search-in-optuna branch from 6ba7852 to 9990aca Compare December 13, 2022 20:54

tenzen-y changed the title ~~[WIP] Support for grid search algorithm in Optuna Suggestion Service~~ Support for grid search algorithm in Optuna Suggestion Service Dec 13, 2022

google-oss-prow bot removed the do-not-merge/work-in-progress label Dec 13, 2022

google-oss-prow bot assigned andreyvelich Dec 13, 2022

google-oss-prow bot added the do-not-merge/hold label Dec 13, 2022

google-oss-prow bot assigned anencore94 and johnugeorge Dec 13, 2022

tenzen-y commented Dec 13, 2022

View reviewed changes

tenzen-y mentioned this pull request Dec 14, 2022

Remove Chocolate Suggestion Service #2058

Closed

3 tasks

anencore94 reviewed Dec 17, 2022

View reviewed changes

tenzen-y force-pushed the support-for-grid-search-in-optuna branch from 2f94b40 to f661f59 Compare December 17, 2022 16:14

andreyvelich reviewed Dec 23, 2022

View reviewed changes

tenzen-y force-pushed the support-for-grid-search-in-optuna branch from e7ac972 to 2cb4d86 Compare December 23, 2022 15:07

tenzen-y commented Dec 23, 2022

View reviewed changes

google-oss-prow bot added the lgtm label Dec 23, 2022

This was referenced Dec 23, 2022

Backend for getting logs of a trial #2039

Merged

Pin the NumPy version with v1.23.5 in some images #2070

Merged

Support for grid search algorithm in Optuna Suggestion Service

339e67f

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

tenzen-y force-pushed the support-for-grid-search-in-optuna branch from 2cb4d86 to 339e67f Compare December 24, 2022 04:23

google-oss-prow bot removed the lgtm label Dec 24, 2022

google-oss-prow bot added the lgtm label Dec 24, 2022

google-oss-prow bot removed the do-not-merge/hold label Dec 24, 2022

google-oss-prow bot merged commit 7c509ba into kubeflow:master Dec 24, 2022

tenzen-y deleted the support-for-grid-search-in-optuna branch December 24, 2022 06:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for grid search algorithm in Optuna Suggestion Service #2060

Support for grid search algorithm in Optuna Suggestion Service #2060

tenzen-y commented Dec 13, 2022 •

edited

Loading

google-oss-prow bot commented Dec 13, 2022

tenzen-y commented Dec 13, 2022

tenzen-y Dec 13, 2022

andreyvelich Dec 23, 2022

anencore94 Dec 17, 2022

tenzen-y Dec 17, 2022 •

edited

Loading

anencore94 Dec 19, 2022

tenzen-y commented Dec 22, 2022

andreyvelich left a comment

andreyvelich Dec 23, 2022

tenzen-y Dec 23, 2022

andreyvelich Dec 23, 2022

tenzen-y commented Dec 23, 2022 •

edited

Loading

tenzen-y Dec 23, 2022

tenzen-y commented Dec 23, 2022

andreyvelich commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

tenzen-y commented Dec 24, 2022

johnugeorge commented Dec 24, 2022

tenzen-y commented Dec 24, 2022

	for _ in range(current_request_number):
	optuna_trial = self.study.ask(fixed_distributions=self._get_optuna_search_space())

	assignments = [Assignment(k, v) for k, v in optuna_trial.params.items()]
	list_of_assignments.append(assignments)

	assignments_key = self._get_assignments_key(assignments)
	self.assignments_to_optuna_number[assignments_key].append(optuna_trial.number)

	logger.Info("Getting suggestions", "endpoint", endpoint, "Number of current request parameters", currentRequestNum, "Number of response parameters", len(responseSuggestion.ParameterAssignments))
	if len(responseSuggestion.ParameterAssignments) != currentRequestNum {
	err := fmt.Errorf("The response contains unexpected trials")
	logger.Error(err, "The response contains unexpected trials")
	return err
	}

Support for grid search algorithm in Optuna Suggestion Service #2060

Support for grid search algorithm in Optuna Suggestion Service #2060

Conversation

tenzen-y commented Dec 13, 2022 • edited Loading

google-oss-prow bot commented Dec 13, 2022

tenzen-y commented Dec 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y Dec 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y commented Dec 22, 2022

andreyvelich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y commented Dec 23, 2022 • edited Loading

Choose a reason for hiding this comment

tenzen-y commented Dec 23, 2022

andreyvelich commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

tenzen-y commented Dec 24, 2022

johnugeorge commented Dec 24, 2022

tenzen-y commented Dec 24, 2022

tenzen-y commented Dec 13, 2022 •

edited

Loading

tenzen-y Dec 17, 2022 •

edited

Loading

tenzen-y commented Dec 23, 2022 •

edited

Loading