MachineSet does not consider current state of Cluster before creating new machines

[ykakarap]

Detailed Description:

When a MachineSet creates machines under certain circumstances, the operation fails or leads to a new machine that will be deleted and recreated in a short timeframe, and this is not ideal.

This issue is about introducing preflight checks (details below) in the MachineSet controller so the machine creation is blocked until circumstances are known to be safe for machine creation, ultimately ensuring stability of workloads in the cluster.

Following cases can be handled more safely by implementing preflight checks:

• When the Kubernetes minor version of the new Machine is greater than control plane minor version, new machine creation might fail (violates Kubernetes version skew)
• When using the KubeadmBootstrapProvider and the Kubernetes minor version of the new Machine is not equal to the control plane minor version, new machine creation might fail (violates the kubeadm version skew)
• When the control plane is upgrading, similar problem as above might happen + other errors might happen if your load balancer is not fast enough in updating its configuration to CP machines being deleted

Like for kubeadm preflight checks, we are considering to have also options to skip preflight checks (probably an annotation, TBD). The main use case for skipping preflight check is to allow users to run a combination of versions outside the kubeadm policy, given that in some cases this works, but it also seems a nice option given the level of configurability we are asked for.

Anything else you would like to add?
Follow-ups:

• Surface preflight check failures in a condition that can be bubbled up to the MachineDeployment.

/kind bug

[fabriziopandini]

/triage accepted

[ykakarap]

/assign

[dlipovetsky]

When using the KubeadmBootstrapProvider and the Kubernetes minor version of the new Machine is not equal to the control plane minor version, new machine creation might fail (violates the kubeadm version skew)

We talked about kubeadm version skew at the Cluster Lifecycle SIG meeting today.

/cc @pacoxu

[dlipovetsky]

(Just for reference, a couple of related issues: #6040, which was closed in favor of #7011)