(3/n) Support 2D Parallelism - Efficient loading of full-state checkpoints #13027

Job	Run time
pl-cpu (macOS-11, lightning, 3.10, 2.0)	25m 46s
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	13m 48s
pl-cpu (windows-2022, lightning, 3.10, 2.0)	17m 57s
pl-cpu (macOS-11, lightning, 3.10, 2.1)	27m 38s
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.1)	14m 0s
pl-cpu (windows-2022, lightning, 3.10, 2.1)	17m 11s
pl-cpu (macOS-11, lightning, 3.10, 2.2)	26m 22s
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.2)	14m 46s
pl-cpu (windows-2022, lightning, 3.10, 2.2)	17m 37s
pl-cpu (macOS-14, lightning, 3.10, 2.3)	14m 39s
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.3)	14m 56s
pl-cpu (windows-2022, lightning, 3.10, 2.3)	18m 32s
pl-cpu (macOS-12, pytorch, 3.11, 2.0)	22m 39s
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.0)	13m 27s
pl-cpu (windows-2022, pytorch, 3.11, 2.0)	17m 57s
pl-cpu (macOS-12, pytorch, 3.11, 2.1)	24m 31s
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.1)	14m 24s
pl-cpu (windows-2022, pytorch, 3.11, 2.1)	18m 9s
pl-cpu (macOS-11, lightning, 3.8, 2.0, oldest)	26m 43s
pl-cpu (ubuntu-20.04, lightning, 3.8, 2.0, oldest)	14m 6s
pl-cpu (windows-2022, lightning, 3.8, 2.0, oldest)	17m 59s
pl-cpu (macOS-11, pytorch, 3.8, 2.0)	24m 40s
pl-cpu (ubuntu-20.04, pytorch, 3.8, 2.0)	14m 10s
pl-cpu (windows-2022, pytorch, 3.8, 2.0)	17m 30s
	7h 29m 27s

Provide feedback