[Tune][CI] Skip zoopt invalid values test #31419
Conversation
Signed-off-by: Justin Yu <justinvyu@berkeley.edu>
…_zoopt_invalid_values_test
| from ray.tune.search.zoopt import ZOOptSearch | ||
|
|
||
| np.random.seed(1000) # At least one nan, inf, -inf and float | ||
| np.random.seed(1002) # At least one nan, inf, -inf and float |
There was a problem hiding this comment.
OOC why're we changing the seed?
There was a problem hiding this comment.
The test was passing for me locally with the old seed. I changed the seed and upped the number of samples to get the test failing on my local machine.
There was a problem hiding this comment.
Does that seed produce one nan, inf, -inf and float?
Which combination makes it fail?
Let's add this as a comment
There was a problem hiding this comment.
Yes, this seed also produces at at least one of each. The failure happens when enough training samples have been collected and then a nan shows up and zoopt tries to replace one of its existing solutions with the nan solution. The previous seed only included a nan solution while it was still collecting training data. Will add as a comment!
| self.skipTest( | ||
| "Recent ZOOpt versions fail handling invalid values gracefully. " | ||
| "Skipping until a fix is added in a future ZOOpt release." | ||
| ) |
There was a problem hiding this comment.
To ensure we eventually re-enable this test, should we create an issue to track this?
| from ray.tune.search.zoopt import ZOOptSearch | ||
|
|
||
| np.random.seed(1000) # At least one nan, inf, -inf and float | ||
| np.random.seed(1002) # At least one nan, inf, -inf and float |
There was a problem hiding this comment.
Does that seed produce one nan, inf, -inf and float?
Which combination makes it fail?
Let's add this as a comment
Signed-off-by: Justin Yu <justinvyu@berkeley.edu>
…_zoopt_invalid_values_test Signed-off-by: Justin Yu <justinvyu@berkeley.edu>
Skips a zoopt searcher test that's causing the `test_searchers` suite in CI to be flaky. Skipping as this is not a Tune issue and needs to be fixed in the zoopt library. I re-enabled this test in #31147 since I thought that nan/inf error handling had been fixed in a recent zoopt release. However, nan/inf values will still cause an error if reported on trial complete. This test should be enabled after polixir/ZOOpt#7 is included in the next zoopt release. Signed-off-by: Justin Yu <justinvyu@berkeley.edu>
Skips a zoopt searcher test that's causing the
test_searcherssuite in CI to be flaky. Skipping as this is not a Tune issue and needs to be fixed in the zoopt library.Why are these changes needed?
I re-enabled this test in #31147 since I thought that nan/inf error handling had been fixed in a recent zoopt release. However, nan/inf values will still cause an error if reported on trial complete.
This test should be enabled after polixir/ZOOpt#7 is included in the next zoopt release.
Related issue number
See #31439 and #31336
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.