fix: Make sure that checkpoints in PBT are removed once they are no longer needed

[mseeger]

… needed

This should solve the issue about checkpoints not being removed. I just return STOP instead of PAUSE. See the code comment why I think this is fine.
I also implemented early checkpoint removal for the synchronous schedulers, which requires the scheduler to flag paused trials for which the checkpoint is no longer needed.

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[mseeger]

I should also check what happens for synchronous multi-fidelity methods.
The potential issues with promo-based ASHA remain, because checkpoints of paused trials are never removed before the end of the experiment.

[mseeger]

Converting this to draft. I'd still like to resoved the situation for synchronous MF schedulers

[codecov]

Codecov Report

Patch coverage: 97.50% and project coverage change: +0.11 🎉

Comparison is base (b4d4ed0) 64.98% compared to head (b514dda) 65.09%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #600      +/-   ##
==========================================
+ Coverage   64.98%   65.09%   +0.11%     
==========================================
  Files         353      354       +1     
  Lines       26238    26279      +41     
==========================================
+ Hits        17051    17107      +56     
+ Misses       9187     9172      -15     

Impacted Files	Coverage Δ
syne_tune/optimizer/schedulers/pbt.py	91.66% <77.77%> (+1.44%)	⬆️
syne_tune/callbacks/remove_checkpoints_callback.py	100.00% <100.00%> (ø)
syne_tune/optimizer/scheduler.py	89.55% <100.00%> (+0.32%)	⬆️
...e/optimizer/schedulers/synchronous/dehb_bracket.py	100.00% <100.00%> (ø)
...tune/optimizer/schedulers/synchronous/hyperband.py	96.75% <100.00%> (+0.20%)	⬆️
...imizer/schedulers/synchronous/hyperband_bracket.py	97.08% <100.00%> (+0.11%)	⬆️
...chedulers/synchronous/hyperband_bracket_manager.py	100.00% <100.00%> (ø)
syne_tune/tuner.py	85.43% <100.00%> (+0.53%)	⬆️
tst/schedulers/test_hyperband_sychronous.py	99.00% <100.00%> (+0.08%)	⬆️
tst/test_pbt.py	90.74% <100.00%> (ø)

... and 6 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[mseeger]

OK, this should go in, since it fixed the checkpoint situation for PBT and the synchronous schedulers. Pre-emptive removal for ASHA promotion is more complicated, I'll do it later.