[MoE] Moe apis #40895

sljlp · 2022-03-24T07:06:52Z

PR types

New features

PR changes

APIs

Describe

add apis about MoE

add _random_routing api in utils add random routing ut

# The first commit's message is: add expert count op add ut for expert_count # This is the 2nd commit message: update UT only for cuda # This is the 3rd commit message: fix for rocm # This is the 4th commit message: update ut # This is the 5th commit message: add moe module # This is the 6th commit message: add expert count op add ut for expert_count # This is the 7th commit message: update UT only for cuda # This is the 8th commit message: update ut # This is the 9th commit message: add moe module # This is the 10th commit message: make expert count private

update utils add limit by capacity op add ut for limit_by_capacity add ut for prune_gate_by_capacity add ut for limit_by_capacity add ut for prune_gate_by_capacity

# The first commit's message is: add expert count op add ut for expert_count # This is the 2nd commit message: update UT only for cuda # This is the 3rd commit message: fix for rocm # This is the 4th commit message: update ut # This is the 5th commit message: add moe module # This is the 6th commit message: add expert count op add ut for expert_count # This is the 7th commit message: update UT only for cuda # This is the 8th commit message: update ut # This is the 9th commit message: add moe module # This is the 10th commit message: make expert count private

paddle-bot-old · 2022-03-24T07:08:16Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

add _random_routing api in utils add random routing ut

ForFishes

LGTM

XiaoguangHu01

LG API

XiaoguangHu01 · 2022-03-29T05:18:29Z

python/paddle/incubate/distributed/models/moe/moe_layer.py

+        fwd_batch_size, )
+
+
+class MoeLayer(nn.Layer):


MoeLayer和MOEGather缩写方式不一致，建议统一采用业界通用的写法。

Ligoml · 2022-03-29T06:27:59Z

python/paddle/incubate/distributed/models/moe/gate/__init__.py

@@ -0,0 +1,18 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.


Ligoml · 2022-03-29T06:28:08Z

python/paddle/incubate/distributed/models/moe/gate/base_gate.py

@@ -0,0 +1,36 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.


Ligoml · 2022-03-29T06:28:22Z

python/paddle/incubate/distributed/models/moe/gate/gshard_gate.py

@@ -0,0 +1,67 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.


Ligoml · 2022-03-29T06:28:36Z

python/paddle/incubate/distributed/models/moe/gate/naive_gate.py

@@ -0,0 +1,37 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.


Ligoml · 2022-03-29T06:28:53Z

python/paddle/incubate/distributed/models/moe/gate/switch_gate.py

@@ -0,0 +1,69 @@
+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.


Ligoml · 2022-03-29T06:55:34Z

python/paddle/incubate/distributed/models/moe/moe_layer.py

+        moe_group: moe group for experts communication
+        mp_group: mp group for mp commutication
+        kwargs: other parameters
+    Examples:


在Args和Examples之间缺乏Returns

Ligoml · 2022-03-29T06:56:42Z

python/paddle/fluid/tests/unittests/test_random_routing_op.py

@@ -0,0 +1,77 @@
+#   Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserved.


Ligoml · 2022-03-29T06:57:02Z

paddle/fluid/operators/prune_gate_by_capacity_op.h

@@ -0,0 +1,33 @@
+// Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.


Ligoml · 2022-03-29T06:57:09Z

paddle/fluid/operators/prune_gate_by_capacity_op.cu

@@ -0,0 +1,123 @@
+// Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.


Ligoml · 2022-03-29T06:57:17Z

paddle/fluid/operators/prune_gate_by_capacity_op.cc

@@ -0,0 +1,123 @@
+// Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.


好的，下个PR会把问题全部修改掉

sljlp and others added 30 commits March 23, 2022 15:28

add random routing op

713bdbe

add _random_routing api in utils add random routing ut

add assign pos op

cda9c51

fix upper num name

7a782d3

add api _assign pos

db110c9

add ut for assign pos op

f310a54

update date

f3ff5f6

add op about moe gate

8968bf1

update utils add limit by capacity op add ut for limit_by_capacity add ut for prune_gate_by_capacity add ut for limit_by_capacity add ut for prune_gate_by_capacity

fix for win

40a8a50

fix bugs in test_limit_by_capacity_op

37950fa

update ut

83a1361

update for test (timeout)

7f704c7

fix ut

7db0bb6

update

441ea29

update(fix) ut for win

b636217

moe apis in incubate

79be77c

add assign pos op

6f46b07

fix upper num name

a300850

add api _assign pos

f8b1852

add ut for assign pos op

c0f2701

update date

2e464c2

fix for win

bb266f8

update for test (timeout)

53e494d

fix ut

3dd05d1

update

d4c0cf5

fix ut for number count

1d19ed0

Merge branch 'assign_pos_op' into limit_by_capacity

d517d5e

Merge branch 'limit_by_capacity' into moe_apis

04f39ef

Merge branch 'random_routing' into moe_apis

f00621b

sljlp added 7 commits March 23, 2022 17:27

add apis and utils

00054fc

add gate apis

a7d9506

add moe and grad clip apis

cfc9fc9

update moe apis

913ec54

add ops for moe gate

e95b2f7

fix

6df2be4

Merge branch 'limit_by_capacity' into moe_apis

d079246

update for base moe layer api

2945176

sljlp added 9 commits March 25, 2022 14:19

add random routing op

0e97c7c

add _random_routing api in utils add random routing ut

fix for dygraph

d94fd57

update with ranodm routing

39926a0

Merge branch 'random_routing' into limit_by_capacity

e56d26c

update

0e65fe8

fix ut for limit by capacity

3bce091

Merge branch 'limit_by_capacity' into moe_apis

3ac3ff2

update

7d49289

Merge branch 'limit_by_capacity' into moe_apis

02f777d

ForFishes approved these changes Mar 26, 2022

View reviewed changes

wanghuancoder approved these changes Mar 28, 2022

View reviewed changes

sljlp changed the title ~~Moe apis~~ [MoE] Moe apis Mar 28, 2022

XiaoguangHu01 approved these changes Mar 29, 2022

View reviewed changes

Ligoml reviewed Mar 29, 2022

View reviewed changes

Ligoml approved these changes Mar 29, 2022

View reviewed changes

ForFishes merged commit aeade53 into PaddlePaddle:develop Mar 29, 2022

sljlp mentioned this pull request Mar 29, 2022

[MoE] Moe apis #41092

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MoE] Moe apis #40895

[MoE] Moe apis #40895

sljlp commented Mar 24, 2022

paddle-bot-old bot commented Mar 24, 2022

ForFishes left a comment

XiaoguangHu01 left a comment

XiaoguangHu01 Mar 29, 2022

sljlp Mar 29, 2022

Ligoml Mar 29, 2022

Ligoml Mar 29, 2022

Ligoml Mar 29, 2022

Ligoml Mar 29, 2022

Ligoml Mar 29, 2022

Ligoml Mar 29, 2022

Ligoml Mar 29, 2022

Ligoml Mar 29, 2022

Ligoml Mar 29, 2022

Ligoml Mar 29, 2022

sljlp Mar 29, 2022

		@@ -0,0 +1,18 @@
		# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.

		@@ -0,0 +1,36 @@
		# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.

		@@ -0,0 +1,67 @@
		# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.

		@@ -0,0 +1,37 @@
		# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.

		@@ -0,0 +1,69 @@
		# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.

		@@ -0,0 +1,77 @@
		# Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserved.

		@@ -0,0 +1,33 @@
		// Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.

		@@ -0,0 +1,123 @@
		// Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.

[MoE] Moe apis #40895

[MoE] Moe apis #40895

Conversation

sljlp commented Mar 24, 2022

PR types

PR changes

Describe

paddle-bot-old bot commented Mar 24, 2022

ForFishes left a comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment