[Feature Request] Adjustment on exploration/exploitation parameter with EI

[roshni-kamath]

🚀 Feature Request

Pardon me if this is already implemented or is due to a misunderstanding of mine, but I am under the impression that there is no way to adjust the exploration/exploitation tradeoff when using the EI acquisition function (in opposition to UCB where there is a beta parameter).

Motivation

When looking at expositions of gaussian optimization and EI acquisition function I see mention of a xi parameter (example: [http://krasserm.github.io/2018/03/21/bayesian-optimization/]) that allows to adjust between exploration/exploitation but I have seen no mention of this in Botorch APIs or documentation...

Pitch

Describe the solution you'd like
A simple adjustable parameter when using optimize_acqf with the EI function.

Thank you

[Balandat]

Not familiar with this formulation, but it seems you can get the equivalent by just absorbing xi into best_f? Like call best_f = f(x*) - xi

[roshni-kamath]

Alright yes, try that, I think it makes sense.

[lcollia]

Hi @Balandat, @roshni-kamath,

I was using GPyOpt for BO and now switched to BOTorch.
I had a tradeoff parameter in GPyOpt for every acquisition function, thus I am interested to conserved this capability.

@roshni-kamath, how did you modified the methods of the ExpectedImprovement class of BOTorch to adjust the exploration/exploitation ratio?

[Balandat]

One immediate way is to just "scale down" the incumbent best_f to induce more exploration as described above.

Btw, we just added a LogExpectedImprovement acquisition function in #1565 that has much better numerical properties (and thus better performance) than classic ExpectedImprovement, so you may want to try that. Note though that this hasn't made it into a release yet and for now is only available on the main branch.

[lcollia]

Thanks for your answers.

The logEI seems very interesting, I will definitively use it when it will be in the next release!

For the modification of best_f, what I don't know is to what extend can I modify it?
If my prediction are between 0 and 1 (thus best_f = 1), and thus the mean of the prediction can be between 0 and 1:

• do I have to decrease the value of best_f independently of the predicted mean (thus I could obtain the situation where mean > modified_best_f),
• or do I have to apply to best_f for each EI calculation a decrease of x% of the difference between mean and best_f (so I always have modified_best_f > mean)

Globally speaking, in GPyOpt all the acquisition functions have an exploration/exploitation ratio. It is not the case for BOTorch (none for EI & PI, but one for UCB). What are the "true" mathematical definition for these acquisition function? Is by definition the "improvement" factor fix (so the uncertainty consideration is always the same for EI and PI), whereas the uncertainty is directly used in UCB so you have to define the sigma to use?

[Balandat]

Globally speaking, in GPyOpt all the acquisition functions have an exploration/exploitation ratio. It is not the case for BOTorch (none for EI & PI, but one for UCB)

Correct, not all acquisition functions in BoTorch have such a parameter.

What are the "true" mathematical definition for these acquisition function? Is by definition the "improvement" factor fix (so the uncertainty consideration is always the same for EI and PI), whereas the uncertainty is directly used in UCB so you have to define the sigma to use?

So the classic definitions of EI and PI don't have an exploration parameter; exploration and exploitation as you say are trade off by way of the variance relative to the mean differences. That said, while EI and PI are Bayesian-optimal for a single step w.r.t the appropriate costs, when used in a BO setting they are really just heuristics. So there is nothing wrong with modifying them and proposing variants of them that do have additional parameters. I'd be hesitant to suddenly add a big zoo of them to BoTorch though. Do you have a pointer to how GPyOpt implements these modified variants?

For the modification of best_f, what I don't know is to what extend can I modify it?

You don't need to have that modified_best_f > mean (what is mean here anyway the maximum posterior mean? The posterior mean at the point corresponding to the true best_f?). Intuitively speaking, as you lower the value of best_f you're making the value of EI across the search space "more similar" and are thus encouraging additional exploration. If you increase best_f you're doing the opposite. If you increase it too much you're likely going to run into numerical issues where EI is zero most everywhere (logEI will help but only so much).

[eytan]

IIRC there are heuristic approaches to adding an exploration bonus to EI, but this isn’t something we’d want to add to BoTorch since there is no theoretical or practical reason to do so. It would certainly be possible to create your own implementation though that does.

…

On Fri, Dec 23, 2022 at 6:59 PM Max Balandat ***@***.***> wrote: Globally speaking, in GPyOpt all the acquisition functions have an exploration/exploitation ratio. It is not the case for BOTorch (none for EI & PI, but one for UCB) Correct, not all acquisition functions in BoTorch have such a parameter. What are the "true" mathematical definition for these acquisition function? Is by definition the "improvement" factor fix (so the uncertainty consideration is always the same for EI and PI), whereas the uncertainty is directly used in UCB so you have to define the sigma to use? So the classic definitions of EI and PI don't have an exploration parameter; exploration and exploitation as you say are trade off by way of the variance relative to the mean differences. That said, while EI and PI are Bayesian-optimal for a single step w.r.t the appropriate costs, when used in a BO setting they are really just heuristics. So there is nothing wrong with modifying them and proposing variants of them that do have additional parameters. I'd be hesitant to suddenly add a big zoo of them to BoTorch though. Do you have a pointer to how GPyOpt implements these modified variants? For the modification of best_f, what I don't know is to what extend can I modify it? You don't need to have that modified_best_f > mean (what is mean here anyway the maximum posterior mean? The posterior mean at the point corresponding to the true best_f?). Intuitively speaking, as you lower the value of best_f you're making the value of EI across the search space "more similar" and are thus encouraging additional exploration. If you increase best_f you're doing the opposite. If you increase it too much you're likely going to run into numerical issues where EI is zero most everywhere (logEI will help but only so much). — Reply to this email directly, view it on GitHub <#373 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAW34M4NLMEIWYJQINUNTDWOY4HZANCNFSM4KUVAHWQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[lcollia]

Thanks both for your answers, it is very helpful!

Do you have a pointer to how GPyOpt implements these modified variants?

Here is the implementation of "EI" in GPyOpt with a jitter value to balance exploration over exploitation: [https://github.com/SheffieldML/GPyOpt/blob/master/GPyOpt/acquisitions/EI.py]

I have to highlight that even the calcul of the improvement look like to me a little different in GPyOpt and in BOTorch

If you increase it too much you're likely going to run into numerical issues where EI is zero most everywhere (logEI will help but only so much).

it is true that I obtain this behavior with the GPyOpt implementation, however I obtain it when I increase the jitter value, so when I go for more exploration. Whereas you mentioned that it will be when going for more exploitation. But GPyOpt do a minimization by default (BOTorch a maximization), so maybe that's the reason of the opposite expected behavior.

this isn’t something we’d want to add to BoTorch since there is no theoretical or practical reason to do so. It would certainly be possible to create your own implementation though that does.

I understand. no pb.

Do you have any reference (paper/book) that state that by default EI/PI don't need explore/exploit ratio?

[eytan]

Hi, Interesting question. I don’t have a great reference here. EI balances exploration and exploitation in a hyperparameter free way just as something like Thompson sampling does. I am not familiar with any recent paper on BayesOpt published in major conferences that includes an exploration bonus. You can also see that highly cited surveys on BayesOpt do not include discussion of exploration bonuses for EI. https://arxiv.org/pdf/1807.02811.pdf https://www.cs.ox.ac.uk/people/nando.defreitas/publications/BayesOptLoop.pdf

…

On Tue, Jan 3, 2023 at 8:33 AM lcollia ***@***.***> wrote: Thanks both for your answers, it is very helpful! Do you have a pointer to how GPyOpt implements these modified variants? Here is the implementation of "EI" in GPyOpt with a jitter value to balance exploration over exploitation: [ https://github.com/SheffieldML/GPyOpt/blob/master/GPyOpt/acquisitions/EI.py ] I have to highlight that even the calcul of the improvement look like to me a little different in GPyOpt and in BOTorch If you increase it too much you're likely going to run into numerical issues where EI is zero most everywhere (logEI will help but only so much). it is true that I obtain this behavior with the GPyOpt implementation, however I obtain it when I increase the jitter value, so when I go for more exploration. Whereas you mentioned that it will be when going for more exploitation. But GPyOpt do a minimization by default (BOTorch a maximization), so maybe that's the reason of the opposite expected behavior. this isn’t something we’d want to add to BoTorch since there is no theoretical or practical reason to do so. It would certainly be possible to create your own implementation though that does. I understand. no pb. Do you have any reference (paper/book) that state that by default EI/PI don't need explore/exploit ratio? — Reply to this email directly, view it on GitHub <#373 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAW34OWKC4Z5MKOQ3SP6ALWQQTB5ANCNFSM4KUVAHWQ> . You are receiving this because you commented.Message ID: ***@***.***>