* init shardformer code structure

* add implement of sharder (inject and replace)

* add implement of replace layer to colossal layer

* separate different layer policy, add some notion

* implement 1d and 2d slicer, can tell col or row

* fix bug when slicing and inject model

* fix some bug; add inference test example

)

* init shardformer code structure

* add implement of sharder (inject and replace)

* add implement of replace layer to colossal layer

* separate different layer policy, add some notion

* implement 1d and 2d slicer, can tell col or row

* fix bug when slicing and inject model

* fix some bug; add inference test example

* add share weight and train example

* add train

* add docstring and readme

* add docstring for other files

* pre-commit

* [shardformer] refactored the user api

* polish code

* update readme with modules content

* remove img

)

* add dropout layer, add dropout test

* modify seed manager as context manager

* add a copy of col_nn.layer

* add dist_crossentropy loss; separate module test

* polish the code

* fix dist crossentropy loss

…3883)

* add gpt2 policy and modify shard and slicer to support

* remove unused code

* polish code

* add bert align test, fix dist loss bug

* forward and backward align

* add ignore index

* add shardformer CI

* add gather_output optional for user in shardconfig

* update readme with optional gather_ouput

* add dist crossentropy loss test, remove unused files

* remove unused file

* remove unused file

* rename the file

* polish code

* fix bug in slicer, add slicer unit test

* add dropout test

* use pid as dropout seed

* updata dropout test with local pattern

* ad todo

#3949)

* add dist dropout in model

* update docstring and bert policy with dropout

* refactor basepolicy and sharded, update bert

* update format

* update gpt2 policy

* update bert policy

* remove unused code

* update readme for new policy usage

adjust layer attr

test t5

* add dist dropout in model

* update docstring and bert policy with dropout

* refactor basepolicy and sharded, update bert

* update format

* update gpt2 policy

* update bert policy

* remove unused code

* update readme for new policy usage

* add downstream model of bert

* remove unused code

* fix an error in readme

* simplify code

* fix an error in readme

* simplify code

* refactor shardformer

* add todo

* remove slicer

* resolve code review

* [shardformer] integrated linear 1D with dtensor

* polish code

)

* [shardformer] refactored embedding and dropout to parallel module

* polish code

* fix bert downstream with new api

* remove comment line

* add gpt2 test and layer class refactor

* add dropout in gpt2 policy

* [shardformer] adapted T5 and LLaMa test to use kit

* polish code

* support kit use for bert test

* support kit test for gpt2

* [shardformer] support module saving and loading

* polish code

* add linearconv1d test

* add linearconv1d test

* add layernorm to bert

* add layernorm test

* add layernorm test with load state dict

* add use_mixedfusedLN in shard config

* refactor policy to support fused_layernorm

* [test] fixed tests failed due to dtensor change

* polish code

* [shardformer] shardformer support opt models

* [shardformer] shardformer support opt models, fix

* [shardformer] shardformer support opt models, fix

* [shardformer] shardformer support opt models, fix

* first v of vit shardformer

* keep vit

* update

* vit shard add vitattention vitlayer

* update num head shard para

* finish test for vit

* add new_model_class & postprocess

* add vit readme

* delete old files & fix the conflict

* fix sth

* [shardformer] add benchmark of shardformer

* [shardformer] add benchmark of shardformer

* [shardformer] refactored some doc and api

* polish code

* [shardformer] made tensor parallelism configurable

* polish code