[Reshard] Implement reshard from s to r with same process_mesh #56039
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
你的PR提交成功，感谢你对开源项目的贡献! |
LiYuRio
force-pushed
the
dev_reshard
branch
3 times, most recently
from
cfa951a
to
f16495d
Compare
LiYuRio
changed the title
LiYuRio
changed the title
ForFishes
reviewed
Aug 9, 2023
| @@ -109,6 +114,21 @@ std::string GetMasterEndpoint() { | |||
| return master_endpoint; | |||
| } | |||
|
|
|||
| std::string GenUniqueCommKey(const std::vector<int64_t>& process_ids) { | |||
There was a problem hiding this comment.
仅仅支持一维process mesh吗?如果高维和低维都是一个process id编号，key是否相同？
There was a problem hiding this comment.
这个函数只负责把传入的process_ids的vector变成唯一的comm_key，有两种情况：
- 如果输入输出的process_mesh相同，这种可以直接调用集合通信操作，只要把展平的process_id传入就行。
- 如果输入输出的process_mesh不同，需要在调用函数前，结合具体情况，分组创建通信组，这时候一般创建的是点对点通信组。
不管是高维还是低维，只要它们参与通信的进程相同，key就是相同的
chenwhql
reviewed
Aug 9, 2023
paddle/phi/core/distributed/auto_parallel/reshard_all_gather_functor.cc
Outdated
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
New features
PR changes
Others
Description
Pcard-73145
支持Shard到Replicate的状态转换，要求：
输入输出的process_mesh为一维；
输出输出不跨mesh；（process_mesh无变化）
输入的shard状态为均匀切分，Tensor的切分维度能被对应组的进程数整除。
以4卡为例，输入输出都是一维process_mesh，[0, 1, 2, 3]，输出为二维Replicate状态，out_dims_mapping = [-1, -1]，输入为二维Shard状态，in_tensor_shape = [4, 8]
TODO：