Give elbow criteria a closer look

[handwerkerd]

Summary

This issue came out of a discussion on #809. A central part of the decision criteria are based on "elbows" when there is a bend in the sorted kappa and rho values. Components with kappa values above the kappa elbow are likely to be accepted and components with rho valeus above the rho elbow are likely to be rejected. When sorted kappa and rho values don't have a clear elbow, non-ideal results can happen. As such there are various ways in which these criteria are shifted be made a bit less aggressive and a huge portion of the decision tree seems to be to deal with edge cases.

One the decision tree part of the code is modularized, we should give a much closer look at how these elbows are calculated and used.

Additional Detail

• Specifically from the discussion in Run getelbow only if there are enough components #809, the kappa threshold is the minimum of an elbow calculated on all kappa values and an elbow calculated on "non-significant" kappa values that are below a kappa threshold. @tsalo pointed out that this means the non-significant kappa elbow should almost always be lower and, since it may include significantly fewer components (currently works with just 2 components) it's not necessarily even an elbow.
• The rho elbow calculation has similar quirks and I (Dan) am starting to realize it may be causing some regular problems I'm seeing. For example, here is the report from the 5 echo data that we've previously highlighted here: https://me-ica.github.io/ohbm-2021-multiecho/five-echo-report/tedana_report.html If you compare the figure below with the one at the report, you can see that modest differences in the components have resulted in more are in the blue "ignored" group vs the green "accepted" group. As annotated below, that's because the rho elbow leaves only 4 components below that elbow threshold. An elbow cut shouldn't be that aggressive.

Next Steps

• Do we agree that this will be much easier to address once the decision tree is modularized [REF] Decision Tree Modularization #756 (i.e. prioritize getting that done before really digging into this)?
  • Additional note is that the minimal decision tree in the proposed modularization is supposed to be a conservative option the doesn't agressively denoise, but it's sometimes very agressive. This is likely because, in the current version, the rho elbow threshold is used to reject components. We'll need to fix this either in the minimal decision tree or by adjusting how the rho elbow is calculated.
• Make sure the kappa and rho elbow values are included in program outputs so we can keep track of them better and understand in which situations issues arise. Possibly includes Adding elbow & variance information to the reports #764
• Systematically look at where elbows are working as expected or failing across a few datasets & see if there is a clear, more conservative way to tweak the elbow calculations.

[tsalo]

Is the rationale behind using Kappa and Rho elbows to select ICA components documented anywhere?

[handwerkerd]

Calculating kappa and rho and the elbows goes back to Prantik's original 2012 pub: https://www.sciencedirect.com/science/article/abs/pii/S1053811911014303
My perspective on this isn't necessary how we calculate an elbow and more that kappa and rho are semi-arbitrary summary metrics for T2* and S0 weighting in each component. The issue is both whether we should assume there is a sufficiently clear elbow for kappa and rho and use elbows as criteria and also whether the kappa & rho summary metrics are robust and informative over a wide range of data.