Support leader-election (HA) for Katib Components

[aaron-arellano]

/kind feature

Describe the solution you'd like
[A clear and concise description of what you want to happen.]
Ability to enable leader-election for Katib components. We run kubeflow in multiple AZs and strive to achieve HA whenever possible for our components to guarantee SLOs for our users. We would like for Katib components to support leader-election to enable HA in the event a node goes down in a particular AZ.

[gaocegege]

SGTM, we should support leader election based on controller-runtime/kubebuilder.

/cc @kubeflow/wg-automl-leads

[tenzen-y]

I'm interested in this feature!
Can I take this? @kubeflow/wg-automl-leads

[johnugeorge]

@tenzen-y Thanks for the interest.
Assigned to you

[gaocegege]

Thanks for your contribution! 🎉 👍