Added single-objective JES with additional conditioning, removal of MO-specific code / additional info-theoretic utils #1738

hvarfner · 2023-03-09T19:58:30Z

variable names consistent with MES/GIBBON. Input constructors for JES & JES-exploit added, as well as optimizer of pathwise posterior samples & utility method for retrieving optima.

Motivation

Refactoring of single-objective JES - larger codebase, but slightly faster, better support and more intuitive (?)

There is a substantial amount of multi-objective specific computation in single-objective JES. At its core, it's relatively similar to MES/GIBBON, and this suggested change tries to reflect this. It runs faster than previously (due to the removal of MO-specific stuff), and is (arguably) a bit more intuitive. Moreover, there is the option of running the two different conditioning variants (noisy or noiseless optimal values). The performance diff, however, is minor.

Also added the exploit-variant of JES for added robustness in cases of model misspecification.

Optimization of samples from the posterior

Added batch optimization of pathwise samples and utility method in acquisition/utils. Hyperparameters are set so as to (anecdotally/empirically) not run into memory or runtime issues on laptop, but still be peak performant.

Input constructors for JES

Support for a Fully Bayesian variant of JES - didn't quite work previously (given that condition_on_observations works with fully bayesian models - see incoming PR!

Have you read the Contributing Guidelines on pull requests?

Yes!

Test Plan

Have tested the related notebooks and Ax interfaces & performance of the two different JES methods.

Related PRs

Yes, proposed fix to FBGPs. Coming soon.

facebook-github-bot · 2023-03-09T19:58:37Z

Hi @hvarfner!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

facebook-github-bot · 2023-03-09T22:04:42Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

saitcakmak · 2023-03-10T07:23:08Z

Hi @hvarfner. Thanks for the PR! I'll try to get this reviewed within the next week. In the mean time: i) We require 100% unit test coverage for BoTorch, so you can work on those. ii) The .py file under tutorials is not needed. We auto generate a .py file from the notebook while uploading the website.

hvarfner · 2023-03-10T07:25:01Z

@saitcakmak Thanks! The .py file in tutorials wasn't supposed to be in there. I'll work on the tests in the coming days!

Balandat · 2023-03-10T14:58:55Z

botorch/utils/sampling.py

+    for iter_ in range(maxiter):
+        # Decaying the learing rate - empirically, this has been well-calibrated
+        lr = lr_init / (iter_ + 1)
+        per_sample_outputs = path_func(X_top_candidates)
+        grads = torch.autograd.grad(
+            per_sample_outputs, X_top_candidates, grad_outputs=torch.ones_like(per_sample_outputs))[0]
+        X_top_candidates = torch.clamp(
+            X_top_candidates + lr * grads, bounds[0], bounds[1])  # TODO fix bounds here


The sample paths here are deterministic (and smooth) functions - rather using a (manual) first-order gradient scheme here, we could probably get much faster convergence by using a (quasi)-second order method such as L-BFGS here.

Sure, will fix!

@Balandat After giving this some more thought, it seems pretty tricky to implement this in batch with the set of tools available. Would a for-loop solution suffice?

I feel like a for loop across each sample would be quite costly. Another option here would be to just define a function that sums the per_sample_outputs and then call L-BFGS-B on that (with the domain being k copies of the original domain, with k the number of sample paths). This is actually what we do for performing acquisition function optimization across random restarts:

botorch/botorch/generation/gen.py

Lines 175 to 203 in 9302055

def f_np_wrapper(x: np.ndarray, f: Callable):

"""Given a torch callable, compute value + grad given a numpy array."""

if np.isnan(x).any():

raise RuntimeError(

f"{np.isnan(x).sum()} elements of the {x.size} element array "

f"`x` are NaN."

)

X = (

torch.from_numpy(x)

.to(initial_conditions)

.view(shapeX)

.contiguous()

.requires_grad_(True)

)

X_fix = fix_features(X, fixed_features=fixed_features)

loss = f(X_fix).sum()

# compute gradient w.r.t. the inputs (does not accumulate in leaves)

gradf = _arrayify(torch.autograd.grad(loss, X)[0].contiguous().view(-1))

if np.isnan(gradf).any():

msg = (

f"{np.isnan(gradf).sum()} elements of the {x.size} element "

"gradient array `gradf` are NaN. "

"This often indicates numerical issues."

)

if initial_conditions.dtype != torch.double:

msg += " Consider using `dtype=torch.double`."

raise RuntimeError(msg)

fval = loss.item()

return fval, gradf

Okay, I did try it some time ago with the torch L-BFGS but I must have done something wrong if it indeed works here. I'll explore it!

Balandat

Not a super thorough review, my main comment is about using a new acquisition function class to implement something generic like epsilon-greedy.

Can you also make sure to run black on this?

Balandat · 2023-03-26T20:10:53Z

botorch/acquisition/joint_entropy_search.py

+
+class qExploitLowerBoundJointEntropySearch(qLowerBoundJointEntropySearch):
+    r"""The acquisition function for the Joint Entropy Search, where the batches
+    `q > 1` are supported through the lower bound formulation.



I don't really like the pattern of defining a whole new acquisition function class for the sole purpose of doing some epsilon-greedy optimization. It seems like this pattern would immediately double the number of acquisition functions if we wanted to do this more generally. I think this sort of notion of how to deal with model misspecification should live at a higher level and the acquisition functions themselves should not be aware of that. For that reason I would prefer we don't add this qExploitLowerBoundJointEntropySearch acquisition function.

Is the main reason you're defining this here so that it's easily useable in Ax via the acquisition function constructor? On the botorch side, this kind of epsilon greedy strategy can easily be implemented by slightly modifying the optimization loop. If this is challenging to do on the Ax side we should make changes there to make that easier (which we're already thinking about - cc @saitcakmak).

Yes, it's for Ax, and I agree. Happy to remove. A middle-ground is to have it as an input constructor & generic acquisition function wrapper GreedyAcquisitionFunction(ActualAcquisitionFunction) otherwise.

I guess that's a separate PR nonetheless.

Balandat · 2023-03-26T20:12:22Z

botorch/acquisition/joint_entropy_search.py


-class qLowerBoundJointEntropySearch(qLowerBoundMultiObjectiveJointEntropySearch):
+
+class qLowerBoundJointEntropySearch(AcquisitionFunction, MCSamplerMixin):


nice, glad we're not making the single-objective acqf a subclass of the multi-objective one any more.

Though it seems a bit odd that qLowerBoundJointEntropySearch is really only the right name if the estimation_type is indeed LB and not MC?

Agreed, changing.

Balandat · 2023-03-26T20:12:48Z

botorch/acquisition/joint_entropy_search.py

+            condition_noiseless:
+            posterior_transform:


Balandat · 2023-03-26T20:19:14Z

botorch/acquisition/joint_entropy_search.py

+        # inputs come as num_optima_per_model x num_models x d
+        # but we want it four-dimensional in the Fully bayesian case,
+        # and three-dimensional otherwise (optionally remove, then re-add)
+        self.optimal_inputs = optimal_inputs.squeeze(1).unsqueeze(-2)
+        self.optimal_outputs = optimal_outputs.squeeze(1).unsqueeze(-2)


Not sure I fully understand what's going on here, can you explain this a bit more? In the non-fully-bayesian setting, I assume num_models is not a dimension, right? In which case the shape would be num_optima_per_model x 1 x d after this logic, but for the fully bayesian model it would be num_optima_per_model x num_models x 1 x d?

Ah I see this is mentioned below. Would be good to have the comment here though.

Sure! Some squeezes/unsqueezes cancelled out, so this should not be as confusing anymore.

botorch/acquisition/joint_entropy_search.py

Balandat · 2023-03-27T01:24:38Z

botorch/utils/sampling.py

+
+    per_sample_outputs = path_func(X_top_candidates).reshape(
+        num_optima * batch_size, num_restarts)
+    sample_argmax = torch.max(per_sample_outputs, dim=-1, keepdims=True)[1].flatten()


Suggested change

sample_argmax = torch.max(per_sample_outputs, dim=-1, keepdims=True)[1].flatten()

sample_argmax = torch.max(per_sample_outputs, dim=-1, keepdims=True).indices.flatten()

Balandat · 2023-03-27T01:27:12Z

botorch/utils/sampling.py

+            which acts as extra initial guesses for the optimization routine.
+        raw_samples: The number of samples with which to query the samples initially.
+        num_restarts The number of gradient descent steps to use on each of the best 
+        found initial points.


Suggested change

found initial points.

found initial points.

Balandat · 2023-03-27T01:27:49Z

botorch/utils/sampling.py

+    argtop_candidates = candidate_queries.argsort(dim=-1, descending=True)[
+        ..., 0:num_restarts]


why not use topk?

Balandat · 2023-03-27T01:28:59Z

botorch/utils/sampling.py

+        ..., 0:num_restarts]
+
+    # These are used as masks when retrieving the argmaxes
+    row_indexer = torch.arange(num_optima * batch_size).to(torch.long)


should be a nullop

Suggested change

row_indexer = torch.arange(num_optima * batch_size).to(torch.long)

row_indexer = torch.arange(num_optima * batch_size)

Balandat · 2023-03-27T01:31:48Z

botorch/utils/sampling.py

+    for iter_ in range(maxiter):
+        # Decaying the learing rate - empirically, this has been well-calibrated
+        lr = lr_init / (iter_ + 1)
+        per_sample_outputs = path_func(X_top_candidates)
+        grads = torch.autograd.grad(
+            per_sample_outputs, X_top_candidates, grad_outputs=torch.ones_like(per_sample_outputs))[0]
+        X_top_candidates = torch.clamp(
+            X_top_candidates + lr * grads, bounds[0], bounds[1])  # TODO fix bounds here


I feel like a for loop across each sample would be quite costly. Another option here would be to just define a function that sums the per_sample_outputs and then call L-BFGS-B on that (with the domain being k copies of the original domain, with k the number of sample paths). This is actually what we do for performing acquisition function optimization across random restarts:

botorch/botorch/generation/gen.py

Lines 175 to 203 in 9302055

def f_np_wrapper(x: np.ndarray, f: Callable):

"""Given a torch callable, compute value + grad given a numpy array."""

if np.isnan(x).any():

raise RuntimeError(

f"{np.isnan(x).sum()} elements of the {x.size} element array "

f"`x` are NaN."

)

X = (

torch.from_numpy(x)

.to(initial_conditions)

.view(shapeX)

.contiguous()

.requires_grad_(True)

)

X_fix = fix_features(X, fixed_features=fixed_features)

loss = f(X_fix).sum()

# compute gradient w.r.t. the inputs (does not accumulate in leaves)

gradf = _arrayify(torch.autograd.grad(loss, X)[0].contiguous().view(-1))

if np.isnan(gradf).any():

msg = (

f"{np.isnan(gradf).sum()} elements of the {x.size} element "

"gradient array `gradf` are NaN. "

"This often indicates numerical issues."

)

if initial_conditions.dtype != torch.double:

msg += " Consider using `dtype=torch.double`."

raise RuntimeError(msg)

fval = loss.item()

return fval, gradf

hvarfner · 2023-03-29T19:58:38Z

@saitcakmak @Balandat There! I hope that it is all in order now. The support for fully Bayesian treatment is there as well, but it doesn't work as of now since the support isn't there. I'll happily keep an eye out to see if everything works as intended once the GPyTorch aspect of it is in place.

facebook-github-bot · 2023-03-29T20:36:54Z

@esantorella has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

codecov · 2023-03-29T20:50:45Z

Codecov Report

Merging #1738 (5a9873e) into main (4d219d6) will not change coverage.
The diff coverage is 100.00%.

❗ Current head 5a9873e differs from pull request most recent head ccd2b1e. Consider uploading reports for the commit ccd2b1e to get more accurate results

@@            Coverage Diff             @@
##              main     #1738    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files          170       170            
  Lines        14783     14912   +129     
==========================================
+ Hits         14783     14912   +129

Impacted Files	Coverage Δ
botorch/acquisition/input_constructors.py	`100.00% <100.00%> (ø)`
botorch/acquisition/joint_entropy_search.py	`100.00% <100.00%> (ø)`
botorch/acquisition/utils.py	`100.00% <100.00%> (ø)`
botorch/utils/sampling.py	`100.00% <100.00%> (ø)`

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Balandat

Thanks! A number of largely nits. I'll take another look at the gpytorch PR for allowing conditioning on samples also in the batched model (fully Bayesian) case in the next few days.

botorch/acquisition/input_constructors.py

botorch/acquisition/joint_entropy_search.py

Balandat · 2023-03-30T05:28:49Z

botorch/acquisition/joint_entropy_search.py

+        cdf_mvs = normal.cdf(normalized_mvs).clamp_min(CLAMP_LB)
+        cdf_rescaled_mvs = normal.cdf(mvs_rescaled_mc).clamp_min(CLAMP_LB)
+        mv_ratio = cdf_rescaled_mvs / cdf_mvs
+
+        log_term = torch.log(mv_ratio) + conditional_logprobs


Hmm it would be good to just work with logcdfs from the get go here for numerical stability, but unfortunately I don't think torch has great support for those

Looks like my past self noticed that as well: pytorch/pytorch#52973

botorch/utils/sampling.py

test/acquisition/test_utils.py

test/utils/test_sampling.py

Balandat · 2023-03-30T05:40:02Z

I'll take another look at the gpytorch PR for allowing conditioning on samples also in the batched model (fully Bayesian) case in the next few days.

For the time being, does your code just throw an error? If so, maybe you can add an informative NotImplementedError pointing to the discussion where we're trying to resolve this? That way we can potentially get this merged in before having everything figured out.

hvarfner · 2023-03-30T15:03:10Z

That makes sense, I added the error as well. I realize that I forgot to mention a specific technical detail previously:

For each of M models, we'd want to condition on an additional N optima (each of which yields a new GP). As such, the (entire) conditional model would have M x N outputs. This works fine for a regular SingleTaskGP, as a batch dimension is simply added when optimal_inputs are of shape N x 1 x D (adds a batch dim, whereas shape N x D just adds N observations to one GP.

The expected behavior in the FB case would then be to have optimal_inputs of shape M x N x 1 x D. I'm not sure if this would work if you want to preserve one copy of the training data only.

Happy to contribute any which way to the FB issue.

hvarfner · 2023-04-01T09:38:41Z

After benchmarking a bit more, the method for obtaining posterior samples (which now uses gen_candidates_scipy) has issues. The quadratic scaling in the number of samples incurred by optimizing their sum (instead of individually) quickly becomes an issue in practice - tens of seconds to find ~64 samples when it should take one.

Personally, I'd like to find adress this issue, either in this PR or in a subsequent one. I think the best action for now is to just go with gen_candidates_torch.

facebook-github-bot · 2023-04-03T22:47:15Z

@esantorella has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

botorch/acquisition/input_constructors.py

esantorella · 2023-04-11T18:49:35Z

Hi, thanks for the pull request! A couple things:

Could you run black and ufmt as described here to fix the errors the linter is throwing?
Could you take a look at the documentation error raised here? It looks like there's an indentation issue. Making the documentation work is finicky, so let me know if you get stuck on that and I'll fix it myself.

hvarfner · 2023-04-12T09:43:57Z

@esantorella Thanks! I want to think that it's fixed now, except for

botorch/botorch/utils/sampling.py

Line 880 in 0fb26d2

paths: SamplePath,

where importing SamplePath causes a circular import. As such, SamplePath remains undefined which flake8 doesn't like.

Balandat

importing SamplePath causes a circular import. As such, SamplePath remains undefined which flake8 doesn't like.

You can get around this with the TYPE_CHECKING var from the typing module, made comments in the code.

botorch/utils/sampling.py

botorch/acquisition/joint_entropy_search.py

variable names consistent with MES/GIBBON. Input constructors for JES & JES-exploit added, as well as optimizer of pathwise posterior samples & utility method for retrieving optima.

Co-authored-by: Max Balandat <Balandat@users.noreply.github.com>

facebook-github-bot · 2023-04-17T12:25:05Z

@esantorella has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

botorch/acquisition/joint_entropy_search.py

tutorials/information_theoretic_acquisition_functions.ipynb

in the example to better highlight differences Changed .forward() to call in optimize_posterior_samples

- Acqf.constructor - Sample optimization - Re-writing of Single-obj JES - Tests - Notebook fitting and changed examples for illustration's sake

optimize_poserior_samples - was accidentally left out of previous commit.

Balandat · 2023-04-21T14:50:32Z

@hvarfner looks like there are a couple of outstanding docs / lint issues. Do you think you can clean that up soon so that we can merge this in?

esantorella · 2023-04-21T16:30:46Z

@hvarfner looks like there are a couple of outstanding docs / lint issues. Do you think you can clean that up soon so that we can merge this in?

These look like merge issues. They were fixed and got un-fixed after merging -- the good news is there is a version that was working.

hvarfner · 2023-04-22T09:06:23Z

@Balandat @esantorella Sorry, I thought I had it figured out last time around. I don't quite understand how I address the last little bit, so I'd appreciate any pointers you may have.

I'm passing tests locally at this point (although flake8 suggests some 40-odd files to be changed) so I assume I'm doing at least one thing incorrectly.

Balandat

Left some comments that should fix the outstanding issues. There are also a couple of lines that aren't covered by tests, please make sure to get those as well. Thanks!

botorch/acquisition/joint_entropy_search.py

tutorials/information_theoretic_acquisition_functions.ipynb

sampling.py from tests.

Balandat · 2023-04-23T15:47:06Z

Looks like we're all good now. @esantorella could you merge this in, please?

facebook-github-bot · 2023-04-23T16:55:12Z

@esantorella has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2023-04-24T13:01:59Z

@esantorella merged this pull request in 04e235d.

Balandat · 2023-04-24T13:28:44Z

Thanks a lot for your contribution, @hvarfner!

hvarfner · 2023-04-24T13:29:46Z

@Balandat @esantorella My pleasure, thanks for your patience! =)

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Mar 9, 2023

Balandat reviewed Mar 10, 2023

View reviewed changes

Balandat requested a review from saitcakmak March 10, 2023 15:00

Balandat reviewed Mar 27, 2023

View reviewed changes

Balandat reviewed Mar 30, 2023

View reviewed changes

hvarfner closed this Apr 1, 2023

hvarfner reopened this Apr 1, 2023

esantorella reviewed Apr 11, 2023

View reviewed changes

botorch/acquisition/input_constructors.py Outdated Show resolved Hide resolved

Balandat reviewed Apr 12, 2023

View reviewed changes

botorch/utils/sampling.py Outdated Show resolved Hide resolved

botorch/utils/sampling.py Show resolved Hide resolved

Balandat reviewed Apr 12, 2023

View reviewed changes

botorch/acquisition/joint_entropy_search.py Outdated Show resolved Hide resolved

hvarfner and others added 7 commits April 13, 2023 22:19

Added single-objective JES with MO-specific computation removed -

ac5b27f

variable names consistent with MES/GIBBON. Input constructors for JES & JES-exploit added, as well as optimizer of pathwise posterior samples & utility method for retrieving optima.

Added changes in constants in JES

30cfc59

Packaged the latest JES changes into one commit

2619667

Bug in sampling testBug in sampling test

46ad5a0

Modified sampling

bb218cf

Formatting, removed imports

5263d8d

Update botorch/utils/sampling.py

8c5a1e4

Co-authored-by: Max Balandat <Balandat@users.noreply.github.com>

esantorella reviewed Apr 17, 2023

View reviewed changes

botorch/acquisition/joint_entropy_search.py Outdated Show resolved Hide resolved

esantorella reviewed Apr 17, 2023

View reviewed changes

tutorials/information_theoretic_acquisition_functions.ipynb Outdated Show resolved Hide resolved

hvarfner added 10 commits April 18, 2023 15:21

Modified tutuorial to fit with new structure, and changed seed

4d562fe

in the example to better highlight differences Changed .forward() to call in optimize_posterior_samples

Addition of all JES changes after rebase

033a565

- Acqf.constructor - Sample optimization - Re-writing of Single-obj JES - Tests - Notebook fitting and changed examples for illustration's sake

Changed citation in JES and forward to __call__ in

011ea93

optimize_poserior_samples - was accidentally left out of previous commit.

Forgot ufmt on sampling

6866d6c

Added fixes to info theoretic tutorial and cleaned up

cabaca3

Added tutorial, sampling and JES reference change

7a1074e

Modified tutorials - finished

8c3cebd

Merge branch 'main' of github.com:pytorch/botorch into tutorial

43a68c6

Matplotlib import disappeared

9d6c638

Merge branch 'tutorial' into main

54d3b42

hvarfner requested a review from esantorella April 22, 2023 09:08

Formatting changes

e543c55

Balandat reviewed Apr 22, 2023

View reviewed changes

botorch/acquisition/joint_entropy_search.py Outdated Show resolved Hide resolved

botorch/acquisition/joint_entropy_search.py Outdated Show resolved Hide resolved

tutorials/information_theoretic_acquisition_functions.ipynb Show resolved Hide resolved

hvarfner and others added 3 commits April 23, 2023 12:09

Added FB test to JES, smoke test to tutorial and excluded one line in

9583008

sampling.py from tests.

Formatting

682b272

Merge branch 'main' into main

ccd2b1e

facebook-github-bot closed this in 04e235d Apr 24, 2023

facebook-github-bot added the Merged label Apr 24, 2023

	def f_np_wrapper(x: np.ndarray, f: Callable):
	"""Given a torch callable, compute value + grad given a numpy array."""
	if np.isnan(x).any():
	raise RuntimeError(
	f"{np.isnan(x).sum()} elements of the {x.size} element array "
	f"`x` are NaN."
	)
	X = (
	torch.from_numpy(x)
	.to(initial_conditions)
	.view(shapeX)
	.contiguous()
	.requires_grad_(True)
	)
	X_fix = fix_features(X, fixed_features=fixed_features)
	loss = f(X_fix).sum()
	# compute gradient w.r.t. the inputs (does not accumulate in leaves)
	gradf = _arrayify(torch.autograd.grad(loss, X)[0].contiguous().view(-1))
	if np.isnan(gradf).any():
	msg = (
	f"{np.isnan(gradf).sum()} elements of the {x.size} element "
	"gradient array `gradf` are NaN. "
	"This often indicates numerical issues."
	)
	if initial_conditions.dtype != torch.double:
	msg += " Consider using `dtype=torch.double`."
	raise RuntimeError(msg)
	fval = loss.item()
	return fval, gradf


		class qLowerBoundJointEntropySearch(qLowerBoundMultiObjectiveJointEntropySearch):

		class qLowerBoundJointEntropySearch(AcquisitionFunction, MCSamplerMixin):

	sample_argmax = torch.max(per_sample_outputs, dim=-1, keepdims=True)[1].flatten()
	sample_argmax = torch.max(per_sample_outputs, dim=-1, keepdims=True).indices.flatten()

		argtop_candidates = candidate_queries.argsort(dim=-1, descending=True)[
		..., 0:num_restarts]

	row_indexer = torch.arange(num_optima * batch_size).to(torch.long)
	row_indexer = torch.arange(num_optima * batch_size)

Added single-objective JES with additional conditioning, removal of MO-specific code / additional info-theoretic utils #1738

Added single-objective JES with additional conditioning, removal of MO-specific code / additional info-theoretic utils #1738

Conversation

hvarfner commented Mar 9, 2023

Motivation

Refactoring of single-objective JES - larger codebase, but slightly faster, better support and more intuitive (?)

Optimization of samples from the posterior

Input constructors for JES

Support for a Fully Bayesian variant of JES - didn't quite work previously (given that condition_on_observations works with fully bayesian models - see incoming PR!

Have you read the Contributing Guidelines on pull requests?

Test Plan

Related PRs

facebook-github-bot commented Mar 9, 2023

Action Required

Process

facebook-github-bot commented Mar 9, 2023

saitcakmak commented Mar 10, 2023

hvarfner commented Mar 10, 2023

Choose a reason for hiding this comment

hvarfner Mar 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Balandat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hvarfner commented Mar 29, 2023 • edited Loading

facebook-github-bot commented Mar 29, 2023

codecov bot commented Mar 29, 2023 • edited Loading

Codecov Report

Balandat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Balandat commented Mar 30, 2023

hvarfner commented Mar 30, 2023

hvarfner commented Apr 1, 2023 • edited Loading

facebook-github-bot commented Apr 3, 2023

esantorella commented Apr 11, 2023

hvarfner commented Apr 12, 2023

Balandat left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Apr 17, 2023

Balandat commented Apr 21, 2023

esantorella commented Apr 21, 2023

hvarfner commented Apr 22, 2023 • edited Loading

Balandat left a comment

Choose a reason for hiding this comment

Balandat commented Apr 23, 2023

facebook-github-bot commented Apr 23, 2023

facebook-github-bot commented Apr 24, 2023

Balandat commented Apr 24, 2023

hvarfner commented Apr 24, 2023

hvarfner Mar 12, 2023 •

edited

Loading

hvarfner commented Mar 29, 2023 •

edited

Loading

codecov bot commented Mar 29, 2023 •

edited

Loading

hvarfner commented Apr 1, 2023 •

edited

Loading

hvarfner commented Apr 22, 2023 •

edited

Loading