[Easy] Run some tutorials only in smoke test mode #1708
Conversation
facebook-github-bot
added
the
CLA Signed
Codecov Report
@@ Coverage Diff @@
## main #1708 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 170 170
Lines 14610 14610
=========================================
Hits 14610 14610 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
@esantorella has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
scripts/run_tutorials.py
Outdated
| IGNORE_SMOKE_TEST_ONLY = { # only used in smoke tests | ||
| "thompson_sampling.ipynb", # very slow without KeOps + GPU | ||
| "composite_mtbo.ipynb", # TODO: very slow, figure out if we can make it faster | ||
| "Multi_objective_multi_fidelity_BO.ipynb", # TODO: very slow, speed up | ||
| } | ||
| IGNORE_STANDARD_ONLY = { |
There was a problem hiding this comment.
I find this naming very confusing. IGNORE_SMOKE_TEST_ONLY means to run it in smoke test only -- maybe we should just call it that RUN_SMOKE_TEST_ONLY?
For IGNORE_STANDARD_ONLY, the comment says "# Causing the tutorials to crash when run without smoke test. Likely OOM.". Does that mean we also run these only in smoke test, in which case they should be in the previous list?
There was a problem hiding this comment.
agreed! This is confusing.
There was a problem hiding this comment.
OK, this should be clearer now. There actually were never any tutorials we ran only in standard mode. I was confused about this and probably so was whoever put those tutorials on the ignore list in the first place, since they actually run fine in smoke test mode. I moved those tutorials to the list now called "RUN_IF_SMOKE_TEST_IGNORE_IF_STANDARD"
|
@esantorella has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@esantorella merged this pull request in 5a215ec. |
Motivation
Some tutorials were being ignored both in "smoke test" and "standard" runs because they OOM or time out in standard runs, but they actually run fine in "smoke test" mode. I moved those three tutorials to the list now called
RUN_IF_SMOKE_TEST_IGNORE_IF_STANDARD(renamed for clarity).Test Plan
These are all running in smoke test mode now (https://github.com/pytorch/botorch/actions/runs/4287842169/jobs/7469160586) and all take less than a minute and an unexceptional amount of memory.
Related PRs
Depends on #1707 (add five-minute timeout for smoke test tutorials)