[Feature Request] Ensemble of optimizers

[ljleb]

Could the extension use multiple optimizers at the same time instead of averaging the results of the scorers? You could have 1 optimizer per payload, or 1 optimizer per image, or an arbitrary number of optimizers maybe?

There needs to be a way of conciliating the optimizer predictions before merging a model I think.

I am not sure this is a good idea, but I wanted to suggest it in case it could enhance the quality of merges.

[WhiteWipe]

Hyperband could be used to conciliate the optimizer predictions. Run multiple optimizers in parallel continuously feeding the results to each other. At every n steps cut the number of optimizers in half, but double the payload number. Simple, but it should work.