Multi-Armed Bandit #343

julianStreibel · 2024-08-14T13:50:24Z

This PR enables bandit models by bringing the following new features:

a class for modeling binary targets
a Beta-Bernoulli multi-armed bandit surrogate class
a Thompson sampling acquisition function (should be turned into a recommender in the future)
a beta prior
a bandit optimization example
the necessary torch infrastructure, such sampler for torch's beta distribution

AVHopp · 2024-08-15T06:55:05Z

@AdrianSosic @julianStreibel is this already ready for review or do the two of you want to do a first round of review first?

AdrianSosic · 2024-08-15T11:17:01Z

@AdrianSosic @julianStreibel is this already ready for review or do the two of you want to do a first round of review first?

Hi @AVHopp, I'll do a first iteration 👍🏼

AdrianSosic

Hi @julianStreibel, thanks so much for this very nice PR. I'm already in the middle of the review/code roast and will continue on Monday. But here already a few things where I need input / that we need to discuss. See ya next week 😎

baybe/recommenders/pure/base.py

baybe/surrogates/multi_armed_bandit.py

examples/Multi_Armed_Bandit/multi_armed_bandit.py

AdrianSosic

Here my open points

baybe/priors/basic.py

baybe/targets/binary.py

baybe/acquisition/acqfs.py

baybe/surrogates/bandit.py

Scienfitz · 2024-08-28T10:56:03Z

the target branch for this PR is not main but dev/surrogates, which however is in the process of being completed/merged. Is this intended?

AdrianSosic · 2024-08-28T10:59:33Z

the target branch for this PR is not main but dev/surrogates, which however is in the process of being completed/merged. Is this intended?

No, that was done by @julianStreibel, probably because the two of haven't talked about where this is supposed to go. Have change it now. Once the dev branch is merged, I'll rebase

AdrianSosic

Here same comment as thread so that we can discuss.

CHANGELOG.md

Scienfitz

hey guys @julianStreibel @AdrianSosic this is fantastic and I love the example - thanks a lot. I might need another round since this is a lot of code, but I'll leave a first round of comments and stupid questions

README.md

baybe/acquisition/base.py

baybe/priors/basic.py

baybe/recommenders/base.py

baybe/recommenders/pure/base.py

examples/Multi_Armed_Bandit/bernoulli_multi_armed_bandit.py

baybe/acquisition/acqfs.py

baybe/surrogates/bandit.py

…tions and recommendations already measured

…ape of 1

AVHopp

Thanks for this epic piece of work :) My main issue is the example which I think is too complex, see the individual comments.

baybe/acquisition/acqfs.py

baybe/acquisition/base.py

baybe/surrogates/bandit.py

baybe/priors/basic.py

examples/Multi_Armed_Bandit/bernoulli_multi_armed_bandit.py

tests/hypothesis_strategies/kernels.py

examples/Multi_Armed_Bandit/bernoulli_multi_armed_bandit.py

baybe/surrogates/bandit.py

AVHopp

Approve to unblock

Switch from positive/negative to success/failure

Switch from True/False to 1.0/0.0

* The normalization factor of the Frobenius norm was wrong. Should have been divided by number of Monte Carlos instead of number of arms. * Computation via bias-variance decomposition is more suggestive.

* Avoids to set the allow_* flags in the RandomRecommender since each run gets a fresh search space instance with untouched metadata * Added comment on remaining necessary flags

Otherwise causes a lot of problems because nan!=nan

julianStreibel requested review from Scienfitz, AdrianSosic and AVHopp as code owners August 14, 2024 13:50

AdrianSosic mentioned this pull request Aug 14, 2024

Bernoulli Target and Bernoulli Multi-Armed Bandit Surrogate #231

Closed

AdrianSosic force-pushed the feature/multi_armed_bandit branch 2 times, most recently from 5ffd6df to 3fe605f Compare August 16, 2024 13:50

AdrianSosic reviewed Aug 16, 2024

View reviewed changes

baybe/recommenders/pure/base.py Outdated Show resolved Hide resolved

baybe/surrogates/multi_armed_bandit.py Outdated Show resolved Hide resolved

examples/Multi_Armed_Bandit/multi_armed_bandit.py Outdated Show resolved Hide resolved

AdrianSosic self-assigned this Aug 21, 2024

AdrianSosic added the new feature New functionality label Aug 21, 2024

AdrianSosic force-pushed the feature/multi_armed_bandit branch from 6cd8fd2 to ea69f52 Compare August 28, 2024 10:41

AdrianSosic reviewed Aug 28, 2024

View reviewed changes

baybe/priors/basic.py Show resolved Hide resolved

baybe/targets/binary.py Outdated Show resolved Hide resolved

baybe/acquisition/acqfs.py Show resolved Hide resolved

baybe/surrogates/bandit.py Show resolved Hide resolved

baybe/surrogates/bandit.py Show resolved Hide resolved

AdrianSosic changed the base branch from dev/surrogates to main August 28, 2024 10:58

AdrianSosic reviewed Aug 28, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

AdrianSosic force-pushed the feature/multi_armed_bandit branch from ea69f52 to 7c6fb72 Compare August 29, 2024 12:11

Scienfitz reviewed Aug 30, 2024

View reviewed changes

julianStreibel and others added 11 commits September 2, 2024 14:02

Add Bernoulli multi-armed bandit surrogate and binary target

1435172

Add multi-armed bandit tests, update fixtures for repeated recommenda…

7662284

…tions and recommendations already measured

Implement Thompson sampling via qSimpleRegret with _default_sample_sh…

9483467

…ape of 1

Add multi-armed bandit examples

cf11f90

Apply pre-commit auto-fixes

ff8cfe4

Fix CONTRIBUTORS.md

4983ec2

Fix __init__.py files

cfa6a44

Rename attributes of BinaryTarget

7417150

Introduce type alias and add attribute type hints

fde7260

Make attributes keyword-only

6045e91

Add missing validators

d635b20

AVHopp reviewed Sep 3, 2024

View reviewed changes

AdrianSosic added 2 commits September 3, 2024 17:27

Extend sufficient statistics docstring

d1600ff

Merge branch 'main' into feature/multi_armed_bandit

30a87d2

AdrianSosic force-pushed the feature/multi_armed_bandit branch 2 times, most recently from 8e1837a to b98b074 Compare September 5, 2024 20:09

AdrianSosic approved these changes Sep 5, 2024

View reviewed changes

AdrianSosic force-pushed the feature/multi_armed_bandit branch from 88e43a3 to 3f5c086 Compare September 5, 2024 20:45

AdrianSosic mentioned this pull request Sep 6, 2024

Dealing with zero-inflated data #368

Open

Scienfitz approved these changes Sep 6, 2024

View reviewed changes

AVHopp approved these changes Sep 6, 2024

View reviewed changes

AdrianSosic force-pushed the feature/multi_armed_bandit branch from ba27028 to b23d400 Compare September 6, 2024 09:45

AdrianSosic and others added 17 commits September 6, 2024 12:06

Rename to BetaBernoulliMultiArmedBanditSurrogate

63f609e

Drop BetaPrior.to_tensor

8839353

Change default values of BinaryTarget to True/False

cd756e9

Change terminology for choice values

cfb133a

Switch from positive/negative to success/failure

Change computational representation of choice values

69a7915

Switch from True/False to 1.0/0.0

Optimize target transformation

9fb4111

Fix MSE calculation

53540f6

* The normalization factor of the Frobenius norm was wrong. Should have been divided by number of Monte Carlos instead of number of arms. * Computation via bias-variance decomposition is more suggestive.

Make binary_targets strategy use different choice values

4fb685c

Use numbers to identify bandit arms

9cf5d50

Reorganize object creation

183623e

* Avoids to set the allow_* flags in the RandomRecommender since each run gets a fresh search space instance with untouched metadata * Added comment on remaining necessary flags

Explictly mention one-hot encoding requirement

783d2dd

Use acquisition function abbreviation

45f2bf6

Change fixture value

fbfcb85

Fix mypy ignore

f4cd63f

Fix set creation

05d56c5

Update CHANGELOG.md

8b9027f

Disallow nan as choice value

8eae563

Otherwise causes a lot of problems because nan!=nan

AdrianSosic force-pushed the feature/multi_armed_bandit branch from b23d400 to 8eae563 Compare September 6, 2024 10:06

AdrianSosic merged commit 99f5e9e into emdgroup:main Sep 6, 2024
9 of 11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi-Armed Bandit #343

Multi-Armed Bandit #343

julianStreibel commented Aug 14, 2024 •

edited by AdrianSosic

Loading

AVHopp commented Aug 15, 2024

AdrianSosic commented Aug 15, 2024

AdrianSosic left a comment

AdrianSosic left a comment

Scienfitz commented Aug 28, 2024

AdrianSosic commented Aug 28, 2024

AdrianSosic left a comment

Scienfitz left a comment

AVHopp left a comment

AVHopp left a comment

Multi-Armed Bandit #343

Multi-Armed Bandit #343

Conversation

julianStreibel commented Aug 14, 2024 • edited by AdrianSosic Loading

AVHopp commented Aug 15, 2024

AdrianSosic commented Aug 15, 2024

AdrianSosic left a comment

Choose a reason for hiding this comment

AdrianSosic left a comment

Choose a reason for hiding this comment

Scienfitz commented Aug 28, 2024

AdrianSosic commented Aug 28, 2024

AdrianSosic left a comment

Choose a reason for hiding this comment

Scienfitz left a comment

Choose a reason for hiding this comment

AVHopp left a comment

Choose a reason for hiding this comment

AVHopp left a comment

Choose a reason for hiding this comment

julianStreibel commented Aug 14, 2024 •

edited by AdrianSosic

Loading