You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe. minerva support for checkpointing an experiment is somewhat patchy. Presently, models can be saved at the end of an experiment, or are cached periodically when using early stopping but there is no dedicated checkpoint support. This would allow users to reload experiments that have stopped for whatever reason (e.g. pre-emption/ timeout on SLURM or an error) and resume where they left off.
Describe the solution you'd like torch already has support for this. It just needs to be implemented into minerva at the appropriate places (most likely in trainer or tasks).
Describe alternatives you've considered hydra has some plugins for this and SLURM message handling but hydra is not yet implemented into minerva.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
minerva
support for checkpointing an experiment is somewhat patchy. Presently, models can be saved at the end of an experiment, or are cached periodically when using early stopping but there is no dedicated checkpoint support. This would allow users to reload experiments that have stopped for whatever reason (e.g. pre-emption/ timeout on SLURM or an error) and resume where they left off.Describe the solution you'd like
torch
already has support for this. It just needs to be implemented intominerva
at the appropriate places (most likely intrainer
ortasks
).Describe alternatives you've considered
hydra
has some plugins for this and SLURM message handling buthydra
is not yet implemented intominerva
.The text was updated successfully, but these errors were encountered: