Fix ssr #559
Open
Fix ssr #559
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Index alignment between coefficient and error histories was thrown off because BaseOptimizer initializes history with an unregularized regression, used in iterative nonconvex regressions. However, SSR does not require an initial guess. This commit removes the initial guess from SSR history_ and adds a test.
Previously, SSR has simply been finding the 0-sparsity model. Instead of the algorithm in the paper, it was executing a Bayes Information Criterion-like model selection, but with a very low l0 penalty. This test asserts #532
Closes #532 This commit implements the inflection point criteria from the SSR paper, although still measuring training loss, rather than cross validation loss.
Also change test to use better-conditioned feature library, since l0 penalty is based upon condition number. Also, fix use of inflection point model selection. Because error history is reversed before calculating inflection index, index is from the end of the list Also, extract calculating the index of inflection in SSR (+ tests)
Also, hack SSR through tests that provide single-feature data in the same way that trapping was handled.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #559 +/- ##
==========================================
+ Coverage 95.30% 95.31% +0.01%
==========================================
Files 37 37
Lines 4046 4059 +13
==========================================
+ Hits 3856 3869 +13
Misses 190 190 ☔ View full report in Codecov by Sentry. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #532. Original SSR implementation only used an l0 penalty which defaulted to 0, meaning the full non-sparse model was always chosen. However, the SSR paper does model selection based upon the "inflection point" of the error history. We implemented the function _ind_inflection which takes as input the error history of the SSR iterations and returns the index at which the inflection occurs, using the metric for inflection described in the SSR paper. This model selection based on error history inflection is the new default when the l0 penalty is 0. The API does not change, but you can note the different results in the unittest
test_ssr_history_selection.