hivedscheduler is prone to misconfig due to daemon Pods, such as weave net and nginx proxy

[hzy46]

After PAI is deployed, run kubectl describe <worker-node>:

This will cause a mis-configuration in hivedscheduler.

[yqwang-ms]

Operator need to pay attention when config Hived to avoid this misconfig:

1. Using kubectl describe nodes to check if these K80 nodes have nearly the same (Allocatable Resources - All Daemon Pods Requests, such as Pods for Device Plugin, Network Plugin, etc), especially for gpu, cpu, memory. If not, please fix it. Assume the aligned minimal resources are: 4 gpus, 23 cpus, and 219GB memory.

Which is mentioned in doc:
https://github.com/microsoft/hivedscheduler/blob/master/doc/user-manual.md#config-quickstart

[hzy46]

Operator need to pay attention when config Hived to avoid this misconfig:

1. Using kubectl describe nodes to check if these K80 nodes have nearly the same (Allocatable Resources - All Daemon Pods Requests, such as Pods for Device Plugin, Network Plugin, etc), especially for gpu, cpu, memory. If not, please fix it. Assume the aligned minimal resources are: 4 gpus, 23 cpus, and 219GB memory.

Which is mentioned in doc:
https://github.com/microsoft/hivedscheduler/blob/master/doc/user-manual.md#config-quickstart

Could hivedscheduler do a pre-check to avoid mis-configuration? Configuring hivedscheduler manually is error-prone (at least for me).

[yqwang-ms]

Yeap. But anyway, it is not just precheck, we need to always check in case this "Allocatable Resources - All Daemon Pods Requests, such as Pods for Device Plugin, Network Plugin, etc" is changed during Hived running.
Operator ensures this, is current the easiest apporach.
Will consider how to auto it later.

[hzy46]

Yeap. But anyway, it is not just precheck, we need to always check in case this "Allocatable Resources - All Daemon Pods Requests, such as Pods for Device Plugin, Network Plugin, etc" is changed during Hived running.
Operator ensures this, is current the easiest apporach.
Will consider how to auto it later.

Yes, exactly.

[yqwang-ms]

Resolved by #4855