Model as Batch Job

[MattToast]

Allows users to launch individual models through an experiment as a batch job with batch settings.

exp = Experiment(...)
rs = exp.craete_run_settings(...)
bs = exp.create_batch_settings(...)
model = exp.create_model("MyModel", rs, batch_settings=bs)
exp.start(model)  # launches the model as a batch job

If a model with batch settings is added to a higher order entity (e.g. an ensemble), the batch settings of the higher order entity will be used and the batch settings of the model will be ignored to avoid an "sbatch-in-sbatch" scenario or similar .

[codecov-commenter]

Codecov Report

Merging #245 (eadcebb) into develop (ff2cf24) will decrease coverage by 0.16%.
The diff coverage is 100.00%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #245      +/-   ##
===========================================
- Coverage    83.90%   83.74%   -0.17%     
===========================================
  Files           59       62       +3     
  Lines         3479     3512      +33     
===========================================
+ Hits          2919     2941      +22     
- Misses         560      571      +11     

Impacted Files	Coverage Δ
smartsim/entity/model.py	96.73% <ø> (-0.04%)	⬇️
smartsim/experiment.py	80.71% <ø> (-0.28%)	⬇️
smartsim/_core/control/controller.py	84.83% <100.00%> (+0.16%)	⬆️
smartsim/_core/control/manifest.py	95.12% <100.00%> (ø)
smartsim/tf/__init__.py	100.00% <0.00%> (ø)
smartsim/tf/utils.py	26.66% <0.00%> (ø)
smartsim/__init__.py	100.00% <0.00%> (ø)

[al-rigazzi]

Looks very good! The only question I have is whether the changes were tested on a Slurm and a PBS system, as the batch mechanism will not be used on Local.

[MattToast]

@al-rigazzi Good Question! I implemented against a slurm system, and I just got a chance to run the new full_wlm test on a PBS system, and both cases the test is passing. Between that an a couple of dummy runs through I did in the repl against both wlms I fairly confident everything is in working order!!

[al-rigazzi]

LGTM -- but let's wait on other reviews.

[ashao]

Looks good to me. Thanks for doing this work!