make WaitCluster smarter.

[medyagh]

Follow up issue after this refactor : #5121

Currently :

• we do "waitForAPIServer"

• we do "WaitForPodsWithLabelRunning" for multiple components:

  • we wait for to pod to be created with no errors.
  • we wait till the created pod status changes from "pending" to "running"

• For all of above waits we wait X minutes for all of the waits regardless of how small or big they be. (in worst case scenario can add up to 30 minutes of waiting till a get a error to user. )

which is super inefficient because some components take milliseconds to come up and some take minutes !
no need to waste minutes of retrying on something that is usually up by miliseconds and also not trying enough for the one that needs minutes. (etcd takes 1.20 minutes, others take 5ms)

Currently also:

We skip all the waiting alltogether, if --wait=false is passed. which is a bit reckless, at least we can wait for pod to be created and let it be pending.

Proposal:

• make the --wait=false option NOT TO SKIP waiting for "checking if pod exist" , but skip checking for "running" (and call pending good enough) .
• choose the timeout based on component. (this will make minikube fail faster when things are wrong and also wait longer when things just need a bit more waiting)

some metrics I copied from other issue #5122

WaitForPodsWithLabelRunning on Hyperkit (mac os)

Name	Time
k8s-app=kube-proxy	9.022185663s	NOT SO BAD
component=etcd	56.515008972s	BAD
component=kube-scheduler	4.369296ms	GOOD :)
component=kube-controller-manager	3.431618ms	GOOD :)
k8s-app=kube-dns	5.533562ms	GOOD:)

WaitForPodsWithLabelRunning on Virtualbox (mac os)

Name	Time
k8s-app=kube-proxy	9.51573752s	NOT SO BAD
component=etcd	1m18.511212486s	BAD
component=kube-scheduler	11.708301ms	GOOD :)
component=kube-controller-manager	43.938549ms	GOOD :)
k8s-app=kube-dns	7.746797ms	GOOD :)

waitForAPIServer

32.352155ms to wait for apiserver status ...

Possible related issues:
#5161

[medyagh]

looking at the kubernetes labels,

$ kc describe pod kube-apiserver -n kube-system
Name:                 kube-apiserver-m5control-plane
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 m5control-plane/172.17.0.2
Start Time:           Thu, 22 Aug 2019 17:06:26 -0700
Labels:               component=kube-apiserver
                      tier=control-plane

I realized we could wait on tier=control-plane instead of waiting on component
or maybe the prirory number ?

first wait for the highest priority then lowest ?

@tstromberg ?

[tstromberg]

Is this still necessary?

[priyawadhwa]

I believe #5757 and #5894 resolve this issue. if not, please reopen @medyagh !