Inducing Point Allocators for Sparse GPs #1652

henrymoss · 2023-02-03T13:50:53Z

Motivation

As requested by @eytan, I have had a go at implementing BO-specific inducing point allocation (IPA) strategies from my new paper (https://arxiv.org/abs/2301.10123). These allow you to build sparse Gaussian processes that are somewhat customized to the particular task at hand, be it BO, active learning e.t.c.

I have rewritten part of your approximate GP code to allow the specification of custom IPA strategies.
BoTorch's existing behaviour (using the greedy variance reduction of Burt et al (2020)) is a special case of the new functionality and remains the default.
I have included one of the new IPAs from my paper to help demonstrate how custom IPAs can be defined. The one I chose is quite complicated (requiring access to model from the previous BO step) and so being able to implement it was a good test case to check that this code for custom IPA initialisation isnt stupid.
There ended up being a lot of IPA related code, so I moved it all into a utility file, with its own testing file.

Test Plan

I have added loads of tests in test_inducing_point_allocators.py. I also added some extra tests higher up in the code in test_approximate_gp.py. There is perhaps a little bit of redundancy between the two sets of tests, but they are testing different interfaces.

Related PRs

I will write a follow-up PR with a notebook demonstrating this new inducing point allocation functionality and showing how a user can define their own IPA.

botorch/acquisition/max_value_entropy_search.py

botorch/models/approximate_gp.py

codecov · 2023-02-03T14:04:51Z

Codecov Report

Merging #1652 (66854e7) into main (5df2fab) will not change coverage.
The diff coverage is 100.00%.

❗ Current head 66854e7 differs from pull request most recent head ca52f7d. Consider uploading reports for the commit ca52f7d to get more accurate results

@@            Coverage Diff            @@
##              main     #1652   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          169       170    +1     
  Lines        14555     14615   +60     
=========================================
+ Hits         14555     14615   +60

Impacted Files	Coverage Δ
botorch/acquisition/max_value_entropy_search.py	`100.00% <ø> (ø)`
botorch/models/approximate_gp.py	`100.00% <100.00%> (ø)`
botorch/models/utils/inducing_point_allocators.py	`100.00% <100.00%> (ø)`

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

facebook-github-bot · 2023-02-03T17:42:05Z

@esantorella has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

esantorella · 2023-02-03T17:44:41Z

Thanks so much for the PR! I've imported it to Meta-internal and someone should be reviewing it soon.

To fix the Sphinx documentation error, could you list the new module here?

Balandat

Thanks for the contribution @henrymoss, this is excellent.

At a high level this all looks good to me, I have two main points, one is about the design of the instanstiation of the quality function for _allocate_inducing_points handled, the other one about avoiding code duplication in _pivoted_cholesky_init. The rest are mostly nits.

botorch/models/approximate_gp.py

botorch/models/utils/inducing_point_allocators.py

test/models/test_approximate_gp.py

botorch/models/utils/inducing_point_allocators.py

esantorella · 2023-02-06T14:20:49Z

botorch/models/approximate_gp.py

@@ -417,131 +448,3 @@ def init_inducing_points(
            var_strat.variational_params_initialized.fill_(0)

        return inducing_points
-
-
-def _select_inducing_points(


Deleting _select_inducing_points is creating an import error for a downstream library (aepsych). Can _select_inducing_points either call one of the new functions or raise a deprecation warning?

I think we probably don't have to get too crazy in terms of deprecations here since this is a "private" method - @crasanders could we just update aepsych to use the new and shiny initialization strategies here?

What did you two decide?

We'll fix the aepsych issue ourselves and then will merge those changes together. @esantorella could you take care of this, please?

Yup I'll put in a PR to aepsych

henrymoss · 2023-02-12T16:35:30Z

Thanks so much for the PR! I've imported it to Meta-internal and someone should be reviewing it soon.

To fix the Sphinx documentation error, could you list the new module here?

Hi @esantorella, I tried to do this but made things worse

sphinx/source/models.rst

botorch/models/approximate_gp.py

test/models/test_approximate_gp.py

sphinx/source/models.rst

esantorella · 2023-02-13T12:22:36Z

Thanks so much for the PR! I've imported it to Meta-internal and someone should be reviewing it soon.
To fix the Sphinx documentation error, could you list the new module here?

Hi @esantorella, I tried to do this but made things worse

Thanks! Looks like there are additionally some citation-related issues. I'll clean those up and make a PR into this branch.

facebook-github-bot · 2023-02-13T12:39:03Z

@esantorella has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

henrymoss · 2023-02-18T15:43:37Z

I think we are ready @Balandat !

Balandat · 2023-02-18T19:33:01Z

Most excellent, thanks for all the cleaning up. We'll just fix the aepsych failure on our end and then we can merge this in!

facebook-github-bot · 2023-02-20T14:24:54Z

@esantorella has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

…Torch function to replacement with equivalent behavior Summary: BoTorch `_select_inducing_points` is being deprecated in pytorch/botorch#1652 / D43001106. It is being replaced by more general inducing point allocator functionality; the legacy behavior remains a special case. Differential Revision: D43438578 fbshipit-source-id: d781b8af2adab4e585c091e1b049c2c5271f832a

…Torch function to replacement with equivalent behavior (facebookresearch#235) Summary: Pull Request resolved: facebookresearch#235 BoTorch `_select_inducing_points` is being deprecated in pytorch/botorch#1652 / D43001106. It is being replaced by more general inducing point allocator functionality; the legacy behavior remains a special case. Reviewed By: Balandat Differential Revision: D43438578 fbshipit-source-id: 0e9945490b3f228b0c2eca584e77bb5f58769023

…Torch function to replacement with equivalent behavior (pytorch#235) Summary: X-link: facebookresearch/aepsych#235 BoTorch `_select_inducing_points` is being deprecated in pytorch#1652 / D43001106. It is being replaced by more general inducing point allocator functionality; the legacy behavior remains a special case. Reviewed By: Balandat Differential Revision: D43438578 fbshipit-source-id: 8682707bc6749157b6c81b73cf70e9e54e5ec26c

facebook-github-bot · 2023-02-24T00:35:30Z

@esantorella has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Summary: ## Motivation As requested by eytan, I have had a go at implementing BO-specific inducing point allocation (IPA) strategies from my new paper (https://arxiv.org/abs/2301.10123). These allow you to build sparse Gaussian processes that are somewhat customized to the particular task at hand, be it BO, active learning e.t.c. 1. I have rewritten part of your approximate GP code to allow the specification of custom IPA strategies. 2. BoTorch's existing behaviour (using the greedy variance reduction of Burt et al (2020)) is a special case of the new functionality and remains the default. 3. I have included one of the new IPAs from my paper to help demonstrate how custom IPAs can be defined. The one I chose is quite complicated (requiring access to model from the previous BO step) and so being able to implement it was a good test case to check that this code for custom IPA initialisation isnt stupid. 4. There ended up being a lot of IPA related code, so I moved it all into a utility file, with its own testing file. X-link: pytorch/botorch#1652 Differential Revision: D43556981 Pulled By: esantorella fbshipit-source-id: 4b3a9364888f4043dde971adfbfebe229e922c46

facebook-github-bot · 2023-02-24T02:49:56Z

@esantorella merged this pull request in d935d10.

Summary: Pull Request resolved: #240 ## Motivation As requested by eytan, I have had a go at implementing BO-specific inducing point allocation (IPA) strategies from my new paper (https://arxiv.org/abs/2301.10123). These allow you to build sparse Gaussian processes that are somewhat customized to the particular task at hand, be it BO, active learning e.t.c. 1. I have rewritten part of your approximate GP code to allow the specification of custom IPA strategies. 2. BoTorch's existing behaviour (using the greedy variance reduction of Burt et al (2020)) is a special case of the new functionality and remains the default. 3. I have included one of the new IPAs from my paper to help demonstrate how custom IPAs can be defined. The one I chose is quite complicated (requiring access to model from the previous BO step) and so being able to implement it was a good test case to check that this code for custom IPA initialisation isnt stupid. 4. There ended up being a lot of IPA related code, so I moved it all into a utility file, with its own testing file. X-link: pytorch/botorch#1652 Test Plan: I have added loads of tests in test_inducing_point_allocators.py. I also added some extra tests higher up in the code in test_approximate_gp.py. There is perhaps a little bit of redundancy between the two sets of tests, but they are testing different interfaces. ## Related PRs I will write a follow-up PR with a notebook demonstrating this new inducing point allocation functionality and showing how a user can define their own IPA. Reviewed By: saitcakmak Differential Revision: D43556981 Pulled By: esantorella fbshipit-source-id: 1fd989c00e6dabf302beec5f4bde0279ccaf158b

henrymoss added 7 commits January 31, 2023 14:10

WIP

b342995

WIP

997600f

WIP

75428c2

WIP

0736875

WIP

15321c0

WIP

778d64a

fixed format

da9e028

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Feb 3, 2023

henrymoss commented Feb 3, 2023

View reviewed changes

botorch/acquisition/max_value_entropy_search.py Show resolved Hide resolved

henrymoss commented Feb 3, 2023

View reviewed changes

botorch/models/approximate_gp.py Show resolved Hide resolved

henrymoss and others added 3 commits February 3, 2023 13:52

Merge branch 'pytorch:main' into henry/IPA

baf6f39

Merge branch 'master' into henry/IPA

c21f037

Merge branch 'henry/IPA' of github.com:henrymoss/botorch into henry/IPA

f727d0e

Balandat reviewed Feb 5, 2023

View reviewed changes

esantorella reviewed Feb 6, 2023

View reviewed changes

botorch/models/utils/inducing_point_allocators.py Show resolved Hide resolved

esantorella reviewed Feb 6, 2023

View reviewed changes

henrymoss added 2 commits February 12, 2023 16:01

Merge branch 'master' into henry/IPA

0c96aec

made suggestions

7cd6b1b

Balandat reviewed Feb 12, 2023

View reviewed changes

sphinx/source/models.rst Outdated Show resolved Hide resolved

henrymoss added 2 commits February 12, 2023 16:48

extra test for 100% coverage

5364c68

maybe fixed docs

3f952d2

Balandat reviewed Feb 12, 2023

View reviewed changes

botorch/models/approximate_gp.py Outdated Show resolved Hide resolved

test/models/test_approximate_gp.py Outdated Show resolved Hide resolved

fix format

7fd3358

Balandat reviewed Feb 12, 2023

View reviewed changes

sphinx/source/models.rst Outdated Show resolved Hide resolved

WIP

4571ec8

henrymoss added 3 commits February 18, 2023 15:16

doc changes

5b4a65a

Merge branch 'master' into henry/IPA

3b4bf38

WIP

252df0e

esantorella mentioned this pull request Feb 20, 2023

Switch aepsych select_inducing_points from soon-to-be-deprecated BoTorch function to replacement with equivalent behavior facebookresearch/aepsych#235

Closed

esantorella self-assigned this Feb 21, 2023

This was referenced Feb 23, 2023

Switch aepsych select_inducing_points from soon-to-be-deprecated BoTorch function to replacement with equivalent behavior (#235) facebookresearch/aepsych#239

Closed

Minor fixes for inducing point allocators #1702

Closed

esantorella and others added 3 commits February 23, 2023 19:23

Merge branch 'main' into henry/IPA

74cbb98

Fix for tensors on different devices error

00fd3d9

Fix for tensors on different devices error

ca52f7d

This was referenced Feb 24, 2023

Inducing Point Allocators for Sparse GPs (#1652) facebookresearch/aepsych#240

Closed

Fix error about tensors on different devices #1701

Closed

facebook-github-bot closed this in d935d10 Feb 24, 2023

facebook-github-bot added the Merged label Feb 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Inducing Point Allocators for Sparse GPs #1652

Inducing Point Allocators for Sparse GPs #1652

henrymoss commented Feb 3, 2023

codecov bot commented Feb 3, 2023 •

edited

Loading

facebook-github-bot commented Feb 3, 2023

esantorella commented Feb 3, 2023

Balandat left a comment

esantorella Feb 6, 2023

Balandat Feb 6, 2023

henrymoss Feb 12, 2023

Balandat Feb 18, 2023

esantorella Feb 20, 2023

henrymoss commented Feb 12, 2023

esantorella commented Feb 13, 2023

facebook-github-bot commented Feb 13, 2023

henrymoss commented Feb 18, 2023

Balandat commented Feb 18, 2023

facebook-github-bot commented Feb 20, 2023

facebook-github-bot commented Feb 24, 2023

facebook-github-bot commented Feb 24, 2023

Inducing Point Allocators for Sparse GPs #1652

Inducing Point Allocators for Sparse GPs #1652

Conversation

henrymoss commented Feb 3, 2023

Motivation

Test Plan

Related PRs

codecov bot commented Feb 3, 2023 • edited Loading

Codecov Report

facebook-github-bot commented Feb 3, 2023

esantorella commented Feb 3, 2023

Balandat left a comment

Choose a reason for hiding this comment

esantorella Feb 6, 2023

Choose a reason for hiding this comment

Balandat Feb 6, 2023

Choose a reason for hiding this comment

henrymoss Feb 12, 2023

Choose a reason for hiding this comment

Balandat Feb 18, 2023

Choose a reason for hiding this comment

esantorella Feb 20, 2023

Choose a reason for hiding this comment

henrymoss commented Feb 12, 2023

esantorella commented Feb 13, 2023

facebook-github-bot commented Feb 13, 2023

henrymoss commented Feb 18, 2023

Balandat commented Feb 18, 2023

facebook-github-bot commented Feb 20, 2023

facebook-github-bot commented Feb 24, 2023

facebook-github-bot commented Feb 24, 2023

codecov bot commented Feb 3, 2023 •

edited

Loading