Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP][Dist] RAF Sharding - Phase 2 #123

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

Tonny-Gu
Copy link
Contributor

Description

(Brief description on what this PR is about)

This PR mainly contains the Expansion Rules and Infer Hints parts.

Checklist

  • PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage
  • Code is well-documented

cc @awslabs/raf-reviewer

commit 4131923
Merge: 61f3dd0 9e36319
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Aug 23 14:28:20 2022 +0000

    Merge branch 'main' of github.com:Tonny-Gu/raf into sharding-pr1

commit 61f3dd0
Merge: dd26fbc 3136651
Author: NekoDaemon <29330054+Tonny-Gu@users.noreply.github.com>
Date:   Mon Aug 8 02:20:36 2022 +0800

    Merge branch 'awslabs:main' into sharding

commit dd26fbc
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Aug 3 18:57:55 2022 +0000

    impl

commit d7deb20
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sun Jul 31 19:10:08 2022 +0000

    impl

commit 6acdac5
Merge: c1ff936 9f26c8e
Author: NekoDaemon <29330054+Tonny-Gu@users.noreply.github.com>
Date:   Mon Aug 1 02:48:04 2022 +0800

    Merge branch 'awslabs:main' into sharding

commit c1ff936
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sun Jul 31 18:46:46 2022 +0000

    impl

commit 4468f2d
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sun Jul 31 09:44:02 2022 +0000

    impl

commit e191d2b
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Jul 19 18:22:34 2022 +0000

    impl

commit 08f8fb0
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Jul 5 22:28:57 2022 +0000

    impl

commit ef45da3
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Jun 28 16:30:07 2022 +0000

    impl

commit 8e9e83e
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Jun 28 08:31:11 2022 +0000

    impl

commit 5394633
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Jun 14 17:52:43 2022 +0000

    fix

commit 65a896d
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Jun 14 17:52:23 2022 +0000

    fix

commit ce663a7
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Jun 14 16:29:39 2022 +0000

    refactor

commit 1727b4f
Merge: f124494 c8ddbc9
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sun Jun 12 16:14:20 2022 +0800

    Merge branch 'awslabs:main' into sharding

commit f124494
Merge: 10749ad f980111
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Fri Apr 22 17:01:58 2022 +0000

    Merge branch 'main' of https://github.com/Tonny-Gu/raf into sharding

commit 10749ad
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Fri Apr 22 17:00:42 2022 +0000

    refactor

commit adf58ec
Merge: ccadefc 4d1d033
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Apr 13 15:38:17 2022 +0000

    Merge branch 'main' of https://github.com/Tonny-Gu/raf into sharding

commit ccadefc
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Mon Apr 11 15:34:48 2022 +0000

    test

commit 7869cf1
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sat Apr 2 12:26:00 2022 +0000

    impl

commit c2b9133
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sat Mar 19 10:09:48 2022 +0000

    refactor

commit 4c80afb
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Fri Mar 18 18:42:43 2022 +0000

    trace shardspec

commit 2133a71
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Fri Mar 18 09:19:49 2022 +0000

    refactor

commit 8a8b783
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Thu Mar 17 16:12:21 2022 +0000

    refactor

commit bd00a6c
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Mar 16 10:21:37 2022 +0000

    lint

commit 8f590b8
Merge: ec654df b87a69d
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Mar 16 10:14:26 2022 +0000

    Merge branch 'multi-comm' of github.com:Tonny-Gu/meta into sharding

commit ec654df
Merge: 0e74b40 1899126
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Mar 16 16:18:34 2022 +0800

    Merge branch 'meta-project:main' into sharding

commit b87a69d
Merge: 4b3ba0a 1899126
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Mar 16 16:18:25 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit 4b3ba0a
Merge: 0e06a41 df740a1
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Mar 15 20:18:54 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit 0e06a41
Merge: c58a495 7afd415
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Fri Mar 11 23:29:36 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit c58a495
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Fri Mar 11 15:29:16 2022 +0000

    fix

commit bd2c3d7
Merge: ad29efa aa423a2
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Mar 9 17:39:40 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit ad29efa
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Mar 9 09:38:52 2022 +0000

    clean

commit 66d2984
Merge: 2df68d2 20eb5a1
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Mar 9 01:11:11 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit 2df68d2
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Mar 8 17:10:37 2022 +0000

    nested tuple

commit b22ba52
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Mar 8 09:25:07 2022 +0000

    fix

commit 0e74b40
Merge: 44214ac 20eb5a1
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Mar 8 16:38:02 2022 +0800

    Merge branch 'meta-project:main' into sharding

commit 995bb1f
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Mar 8 08:37:16 2022 +0000

    nested tuple

commit eefd9eb
Merge: 5407151 279dba8
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sat Mar 5 14:59:25 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit 5407151
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sat Mar 5 06:56:58 2022 +0000

    clean

commit 44214ac
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Mar 1 14:47:15 2022 +0000

    fix

commit dfc60f2
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Mar 1 12:32:16 2022 +0000

    merge

commit 86b4e69
Merge: 46eec3e df572d2
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Mar 1 12:22:52 2022 +0000

    Merge branch 'main' of github.com:Tonny-Gu/meta into sharding

commit c924e80
Merge: efea2ff df572d2
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sun Feb 27 22:52:29 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit efea2ff
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Fri Feb 25 15:06:46 2022 +0000

    fix

commit 43b2100
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Fri Feb 25 14:59:30 2022 +0000

    merge

commit 4f95e2a
Merge: 1dbe464 654a121
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Fri Feb 25 14:48:24 2022 +0000

    Merge branch 'main' of github.com:Tonny-Gu/meta into multi-comm

commit 46eec3e
Merge: 5989fe1 7e595b5
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Feb 22 13:29:35 2022 +0000

    Merge branch 'main' of github.com:Tonny-Gu/meta into sharding

commit 1dbe464
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Feb 22 13:12:54 2022 +0000

    clean

commit 6e8a865
Merge: a4deaed 7e595b5
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Feb 22 19:56:18 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit a4deaed
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sat Feb 19 08:26:22 2022 +0000

    fix

commit 1bff858
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sat Feb 19 08:01:51 2022 +0000

    license

commit 9032d21
Merge: 8e02780 8a941b8
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sat Feb 19 07:51:56 2022 +0000

    Merge branch 'main' of github.com:Tonny-Gu/meta into multi-comm

commit 8e02780
Merge: 1f994f0 e736065
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sat Feb 19 07:41:58 2022 +0000

    Merge branch 'comm-refactor' of github.com:Tonny-Gu/meta into multi-comm

commit e736065
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sat Feb 19 07:39:41 2022 +0000

    fix

commit 31e20ff
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Fri Feb 11 15:13:46 2022 +0000

    refactor

commit ec73ffd
Merge: 3395d68 aa3b2a8
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Fri Feb 11 14:51:48 2022 +0800

    Merge branch 'meta-project:main' into comm-refactor

commit 1f994f0
Merge: 3395d68 aa3b2a8
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Fri Feb 11 14:51:35 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit 3395d68
Merge: e9ede20 d0da258
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Thu Feb 10 17:52:48 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit e9ede20
Merge: ef2138e ba516c2
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Feb 9 22:13:06 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit ef2138e
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Feb 9 13:23:02 2022 +0000

    fix

commit 967ab79
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Feb 9 12:09:54 2022 +0000

    fix

commit 026fc34
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Feb 9 04:06:38 2022 +0000

    lint

commit b62320b
Merge: b676af6 08f6ffa
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Feb 9 11:58:16 2022 +0800

    Merge branch 'meta-project:main' into multi-comm

commit b676af6
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Feb 9 03:57:25 2022 +0000

    fix gpu test

commit 6b1bc9b
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Feb 8 15:27:46 2022 +0000

    fix cpu test

commit 4c5b083
Merge: bccc991 41fc64e
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Feb 8 10:15:11 2022 +0000

    Merge branch 'main' of github.com:Tonny-Gu/meta into multi-comm

commit bccc991
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Jan 4 07:01:26 2022 +0000

    lint

commit e4d256e
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Jan 4 06:49:18 2022 +0000

    lint

commit 3f03bff
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Mon Jan 3 16:37:44 2022 +0000

    lint

commit 8d8d898
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Mon Jan 3 13:59:26 2022 +0000

    implement

commit 230828d
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Mon Jan 3 13:20:55 2022 +0000

    implement

commit a671d38
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Mon Jan 3 10:31:47 2022 +0000

    implement

commit af7b85b
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sun Jan 2 17:07:56 2022 +0000

    test & lint

commit b40e9d1
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sun Jan 2 16:33:22 2022 +0000

    implement

commit a8b875e
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sat Jan 1 15:13:12 2022 +0000

    test

commit b93a6ab
Merge: e6548da 6215bfc
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sat Jan 1 15:02:43 2022 +0000

    Merge branch 'master' of github.com:Tonny-Gu/meta into multi-comm

commit e6548da
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sat Jan 1 14:23:57 2022 +0000

    implement

commit 47701b5
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sun Dec 19 12:27:35 2021 +0000

    implement

commit e45b60a
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Dec 14 08:19:30 2021 +0000

    implement

commit 56409b7
Merge: c8c7381 5ca2012
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Mon Dec 13 14:56:37 2021 +0800

    Merge branch 'meta-project:master' into multi-comm

commit c8c7381
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Mon Dec 13 06:54:48 2021 +0000

    update

commit 97f5aac
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Sat Dec 11 08:21:32 2021 +0000

    update

commit 5d1a7b9
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Fri Dec 10 16:12:34 2021 +0000

    fix

commit 5989fe1
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Nov 9 06:25:30 2021 +0000

    merge

commit 61ae89b
Merge: 4bbd779 5114fb6
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Nov 9 06:24:05 2021 +0000

    Merge branch 'master' of github.com:Tonny-Gu/meta into sharding

commit 4bbd779
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Nov 9 05:57:41 2021 +0000

    refactor

commit 3e2a3ae
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Fri Oct 29 03:45:31 2021 +0000

    run!

commit 11d08d2
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Oct 26 13:40:00 2021 +0000

    implement

commit cf306f7
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Oct 19 16:13:52 2021 +0000

    implement

commit 527141f
Merge: 8e4271e eae1132
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Oct 19 00:54:47 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit 8e4271e
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Mon Oct 18 16:53:41 2021 +0000

    implement

commit c22c5a0
Merge: 75ddb68 c25a152
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Thu Oct 14 05:47:56 2021 +0000

    Merge branch 'master' of github.com:Tonny-Gu/meta into sharding

commit 75ddb68
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Thu Oct 14 05:45:48 2021 +0000

    implement

commit 971c93e
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Sep 28 09:05:02 2021 +0000

    refactor

commit c67fbdf
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Sep 28 07:49:53 2021 +0000

    implement

commit c676ee5
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Sep 22 15:57:38 2021 +0000

    implement

commit 9b30495
Merge: dea0149 b5845ae
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Sep 22 15:16:01 2021 +0000

    Merge branch 'master' of github.com:Tonny-Gu/meta into sharding

commit dea0149
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Sep 15 13:05:12 2021 +0000

    implement

commit c28de27
Merge: 6137b98 caa3650
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Mon Sep 13 22:41:22 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit 6137b98
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Mon Sep 13 14:39:51 2021 +0000

    implement

commit 7587a1f
Author: Tonny-Gu <Cubies@vip.qq.com>
Date:   Sat Sep 11 13:51:37 2021 +0000

    implement

commit 7315d93
Author: Tonny-Gu <Cubies@vip.qq.com>
Date:   Fri Sep 10 03:58:02 2021 +0000

    fix bug

commit b26566a
Merge: 7d0a519 8b24b09
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Thu Sep 9 14:44:45 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit 7d0a519
Author: Tonny-Gu <Cubies@vip.qq.com>
Date:   Thu Sep 9 06:43:40 2021 +0000

    refactor and fix bug

commit 25bb65d
Author: Tonny-Gu <Cubies@vip.qq.com>
Date:   Tue Sep 7 09:41:08 2021 +0000

    implement

commit deb3d43
Author: Tonny-Gu <Cubies@vip.qq.com>
Date:   Mon Sep 6 03:52:12 2021 +0000

    fix bug

commit a08fd84
Merge: af6df28 469357d
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Mon Sep 6 11:24:59 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit af6df28
Merge: ebf6dc6 08dcf2e
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Mon Aug 30 15:12:02 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit ebf6dc6
Merge: 94f90ba 0110e38
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Fri Aug 27 15:17:31 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit 94f90ba
Merge: ec43e77 c87ae8e
Author: Tonny-Gu <Cubies@vip.qq.com>
Date:   Thu Aug 26 09:27:10 2021 +0000

    Merge remote-tracking branch 'origin/master' into sharding

commit ec43e77
Author: Tonny-Gu <Cubies@vip.qq.com>
Date:   Thu Aug 26 09:21:44 2021 +0000

    implement

commit 50a90f9
Author: Tonny-Gu <Cubies@vip.qq.com>
Date:   Mon Aug 23 11:34:13 2021 +0000

    implement

commit 4a50d9d
Author: Tonny-Gu <Cubies@vip.qq.com>
Date:   Sat Aug 21 08:04:06 2021 +0000

    extend shardspec family

commit 38d46bd
Merge: 595cd9c 3ed4598
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Thu Aug 19 17:48:56 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit 595cd9c
Author: Tonny-Gu <Cubies@vip.qq.com>
Date:   Thu Aug 19 09:23:58 2021 +0000

    implement

commit 637d4aa
Merge: 77ad1b2 f6f87a3
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Wed Aug 18 14:47:41 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit 77ad1b2
Merge: 3de8293 e0e003e
Author: NekoDaemon <Gutongzhou@gmail.com>
Date:   Tue Aug 17 18:13:44 2021 +0800

    Merge branch 'meta-project:master' into sharding

commit 3de8293
Author: Tonny-Gu <Cubies@vip.qq.com>
Date:   Tue Aug 17 10:09:42 2021 +0000

    implement

commit d8cf092
Author: Tonny-Gu <Cubies@vip.qq.com>
Date:   Mon Aug 16 12:32:30 2021 +0000

    add ShardSpec, ShardOpAttrs, InitShardOpAttrs Pass
@Tonny-Gu
Copy link
Contributor Author

@hgt312 @comaniac This PR is ready except for the unit test part.

Pass ExpandShardOpCall();

/*!
* \brief .
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing doc


namespace shard_pass {

class ShardOpCallAttrsSetter : public ExprMutator {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider using mixedmodemutator/exprrewriter for these passes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants