[#133] feat(netty): integration-test supports netty. #1008
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jerqi
reviewed
Jul 12, 2023
| @@ -252,7 +252,9 @@ public <K, V, C> ShuffleHandle registerShuffle(int shuffleId, int numMaps, Shuff | |||
| // get all register info according to coordinator's response | |||
| Set<String> assignmentTags = RssSparkShuffleUtils.getAssignmentTags(sparkConf); | |||
| ClientUtils.validateClientType(clientType); | |||
| assignmentTags.add(clientType); | |||
| if (!sparkConf.get(RssSparkConfig.RSS_TEST_MODE_ENABLE)) { | |||
There was a problem hiding this comment.
Why do we need this? We should reduce similar code. If we don't have better solution, it's acceptable.
|
You can run the command to fix code style issues |
Codecov Report
@@ Coverage Diff @@
## master #1008 +/- ##
============================================
+ Coverage 54.76% 55.19% +0.43%
- Complexity 2526 2545 +19
============================================
Files 362 362
Lines 19334 19377 +43
Branches 1799 1803 +4
============================================
+ Hits 10588 10695 +107
+ Misses 8116 8055 -61
+ Partials 630 627 -3
... and 6 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
jerqi
reviewed
Jul 24, 2023
| @@ -105,9 +105,8 @@ public Map runTest(SparkSession spark, String fileName) throws Exception { | |||
| map = javaPairRDD.collectAsMap(); | |||
| shufflePath = appPath + "/1"; | |||
| assertTrue(new File(shufflePath).exists()); | |||
| } else { | |||
| runCounter++; | |||
There was a problem hiding this comment.
When running rssWithNetty, the path with shuffleId=0 has been cleaned up, and an assertion exception will be thrown at this time, assertTrue(fs.exists(new Path(shufflePath)));
There was a problem hiding this comment.
@zuston Could you take a look at this place?
There was a problem hiding this comment.
I just keep origin code logic.
leixm
commented
Jul 24, 2023
| @@ -150,4 +152,8 @@ public String toString() { | |||
|
|
|||
| return sb.toString(); | |||
| } | |||
|
|
|||
| public synchronized void copyDataTo(ByteBuf to) { | |||
There was a problem hiding this comment.
ReplicateWrite would cause concurrency decode, that's why we added synchronized copyDataTo here.
jerqi
approved these changes
Jul 24, 2023
This was referenced Feb 10, 2024
Closed
jerqi
pushed a commit
that referenced
this pull request
Feb 26, 2024
### What changes were proposed in this pull request? Fix [#1008](#1008). It does not actually test `GRPC_NETTY` mode, because it uses `ShuffleServerGrpcClient` everywhere instead of `ShuffleServerGrpcNettyClient`. Setting the shuffle server's tags to `GRPC_NETTY,GRPC` is useless, because we are not using `ShuffleServerGrpcNettyClient` at all. ### Why are the changes needed? It is a sub PR for: #1519 Also, it is a follow-up PR for: #1008 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing UTs.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
For #133
Why are the changes needed?
Make integration-test support netty.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
UTs.