-
Notifications
You must be signed in to change notification settings - Fork 775
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update katib-config document to describe suggestion service images #1907
Update katib-config document to describe suggestion service images #1907
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for this PR, I left few comments.
Also, can you update about algorithms implementation here: https://www.kubeflow.org/docs/components/hyperparameter-tuning/experiment/#random-search.
For example, for Random we say that Katib uses only hyperopt
implementation.
@@ -114,7 +114,13 @@ All of these settings except **`image`** can be omitted. If you don't specify an | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should merge this PR after we update Katib Manifests images with the latest images.
/cc @johnugeorge
**Must be specified**. | ||
**Must be specified**. You can specify following images: | ||
|
||
- `suggestion-chocolate`: [chocolate](https://github.com/AIworx-Labs/chocolate) based suggestion service which supports `grid`, `chocolate-random`, `chocolate-quasirandom`, `chocolate-bayesian-optimization` and `chocolate-mocmaes`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe it is better to say: "Which supports algorithm names: grid
, ..." ?
And do we need to say about each algorithm name. For example:
chocolate-quasirandom
: Chocolate implementation of quasirandom algorithm.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I mind two things about this:
- In the discussion of Add Goptuna based suggestion service for CMA-ES. katib#1131 (comment), we decided to not introduce
goptuna-random
andgoptuna-tpe
. So I don't know how should I describetpe
andrandom
. - I think it's better to describe
image
field of suggestion settings in this section. How about describe algorithm name in other parts?
|
||
- `suggestion-chocolate`: [chocolate](https://github.com/AIworx-Labs/chocolate) based suggestion service which supports `grid`, `chocolate-random`, `chocolate-quasirandom`, `chocolate-bayesian-optimization` and `chocolate-mocmaes`. | ||
- `suggestion-goptuna`: [Goptuna](https://github.com/c-bata/goptuna) based suggestion service which supports `cmaes`, `tpe` and `random`. | ||
- `suggestion-hyperband`: [HpBandSter](https://github.com/automl/HpBandSter) based suggestion service which supports `hyperband`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe for hyperband we are using our own implementation. We can add link to Katib hyperband source code here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your comment. Hm, we may change the file path of hyperband source code at kubeflow/katib#1144.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, we will change the file path for hyperband. In this PR maybe just add link to hyperband service: https://github.com/kubeflow/katib/blob/master/pkg/suggestion/v1alpha3/hyperband_service.py?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
I believe the merge conflict on this PR is due to PR #1909, which inserted the |
cb75738
to
cb9cd29
Compare
@@ -181,7 +181,8 @@ sampling without replacement. Random search is therefore the best algorithm to | |||
use when combinatorial exploration is not possible. If the number of continuous | |||
variables is high, you should use quasi random sampling instead. | |||
|
|||
Katib uses the [hyperopt](http://hyperopt.github.io/hyperopt/) optimization | |||
Katib uses the [hyperopt](http://hyperopt.github.io/hyperopt/) or |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have no good idea how describe chocolate-random
and chocolate-quasirandom
. So I added Goptuna only.
It may be better to rename chocolate-random
to random
for consistency (related discussion: kubeflow/katib#1131 (comment)).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with @c-bata. What do you think @gaocegege @johnugeorge ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SGTM!
/assign @andreyvelich |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
@sarahmaddox I think we can merge this PR. We will update manifest with the recent Katib version soon.
/cc @johnugeorge @gaocegege
/cc @jlewi Thanks @andreyvelich. I'm no longer working on Kubeflow, so I've cc-ed @jlewi to approve this PR. |
/approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jlewi The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Update katib-config document to describe suggestion service images
Katib issue: kubeflow/katib#1147
cc: @gaocegege @andreyvelich @sarahmaddox