Add Deepspeed Zero 3 MiCS support (Issues #20378) #9802

Job	Run time
fabric-cpu (macOS-14, lightning, 3.10, 2.1)	8m 52s
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.1)	6m 20s
fabric-cpu (windows-2022, lightning, 3.10, 2.1)	8m 9s
fabric-cpu (macOS-14, lightning, 3.11, 2.2.2)	8m 38s
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.2.2)	5m 55s
fabric-cpu (windows-2022, lightning, 3.11, 2.2.2)	6m 52s
fabric-cpu (macOS-14, lightning, 3.11, 2.3)	8m 39s
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.3)	6m 24s
fabric-cpu (windows-2022, lightning, 3.11, 2.3)	7m 39s
fabric-cpu (macOS-14, lightning, 3.12, 2.4.1)	9m 51s
fabric-cpu (ubuntu-22.04, lightning, 3.12, 2.4.1)	8m 5s
fabric-cpu (windows-2022, lightning, 3.12, 2.4.1)	7m 53s
fabric-cpu (macOS-14, lightning, 3.12, 2.5.1)	9m 9s
fabric-cpu (ubuntu-22.04, lightning, 3.12, 2.5.1)	8m 36s
fabric-cpu (windows-2022, lightning, 3.12, 2.5.1)	9m 22s
fabric-cpu (macOS-14, fabric, 3.12, 2.5.1)	9m 27s
fabric-cpu (ubuntu-22.04, fabric, 3.12, 2.5.1)	8m 36s
fabric-cpu (windows-2022, fabric, 3.12, 2.5.1)	8m 3s
fabric-cpu (macOS-14, lightning, 3.9, 2.1, oldest)	7m 26s
fabric-cpu (ubuntu-20.04, lightning, 3.9, 2.1, oldest)	5m 1s
fabric-cpu (windows-2022, lightning, 3.9, 2.1, oldest)	6m 50s
fabric-cpu (macOS-14, fabric, 3.9, 2.1)	8m 45s
fabric-cpu (ubuntu-20.04, fabric, 3.9, 2.1)	5m 38s
fabric-cpu (windows-2022, fabric, 3.9, 2.1)	6m 37s
	3h 6m 47s

Provide feedback