-
Notifications
You must be signed in to change notification settings - Fork 148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Serious Bug] Inaccurate flow control leads to Shuffle server OOM when enabling Netty #1472
Closed
3 tasks done
Comments
3 tasks
This only affects off heap Netty? |
Currently, it only affects off heap Netty. |
rickyma
changed the title
[Serious Bug] Inaccurate flow control leads to Shuffle server OOM
[Serious Bug] Inaccurate flow control leads to Shuffle server OOM when enabling Netty
Jan 31, 2024
Progress: I already have a version that can solve this problem, and I am still testing it. |
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 5, 2024
…erver OOM when enabling Netty
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 5, 2024
…erver OOM when enabling Netty
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 5, 2024
…erver OOM when enabling Netty
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 5, 2024
…erver OOM when enabling Netty
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 5, 2024
…erver OOM when enabling Netty
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 5, 2024
…erver OOM when enabling Netty
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 5, 2024
…erver OOM when enabling Netty
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 5, 2024
…erver OOM when enabling Netty
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 5, 2024
…erver OOM when enabling Netty
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 5, 2024
…erver OOM when enabling Netty
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 5, 2024
…erver OOM when enabling Netty
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 5, 2024
…erver OOM when enabling Netty
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 5, 2024
…erver OOM when enabling Netty
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 5, 2024
…erver OOM when enabling Netty
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 12, 2024
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 12, 2024
…e blocks instead of reallocating it
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 12, 2024
…ntException issues in extremely rare scenarios
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 12, 2024
…e blocks instead of reallocating it
jerqi
pushed a commit
that referenced
this issue
Feb 13, 2024
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 13, 2024
…DirectMemory and usedDirectMemory
jerqi
pushed a commit
that referenced
this issue
Feb 14, 2024
…Memory and usedDirectMemory (#1524) ### What changes were proposed in this pull request? We need to know the exact direct memory usage of `PooledByteBufAllocator` in `Netty`. So we should introduce metrics for Netty's `pinnedDirectMemory` and `usedDirectMemory`. ### Why are the changes needed? A sub PR for: #1519 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing UTs.
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 14, 2024
…ntException issues in extremely rare scenarios
jerqi
pushed a commit
that referenced
this issue
Feb 15, 2024
…ption issues in extremely rare scenarios (#1522) ### What changes were proposed in this pull request? Improve the robustness of methods `ShuffleDataResult.release()` and `ShuffleIndexResult.release()` to fix occasional IllegalReferenceCountException issues in extremely rare scenarios. ### Why are the changes needed? A sub PR for: #1519 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing UTs.
jerqi
pushed a commit
that referenced
this issue
Feb 15, 2024
…ks instead of reallocating it (#1521) ### What changes were proposed in this pull request? Reuse ByteBuf when decoding shuffle blocks instead of reallocating it ### Why are the changes needed? A sub PR for: #1519 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing UTs.
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 21, 2024
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 21, 2024
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 21, 2024
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 22, 2024
…on issues when exceptions happened in clientReadHandler.readShuffleData()
zuston
pushed a commit
that referenced
this issue
Feb 23, 2024
…mory issue causing OOM (#1534) ### What changes were proposed in this pull request? When we use `UnpooledByteBufAllocator` to allocate off-heap `ByteBuf`, Netty directly requests off-heap memory from the operating system instead of allocating it according to `pageSize` and `chunkSize`. This way, we can obtain the exact `ByteBuf` size during the pre-allocation of memory, avoiding distortion of metrics such as `usedMemory`. Moreover, we have restored the code submission of the PR [#1521](#1521). We ensure that there is sufficient direct memory for the Netty server during decoding `sendShuffleDataRequest` by taking into account the `encodedLength` of `ByteBuf` in advance during the pre-allocation of memory, thus avoiding OOM during decoding `sendShuffleDataRequest`. Since we are not using `PooledByteBufAllocator`, the PR [#1524](#1524) is no longer needed. ### Why are the changes needed? A sub PR for: #1519 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing UTs.
zuston
pushed a commit
that referenced
this issue
Feb 24, 2024
…ler.readShuffleData (#1536) ### What changes were proposed in this pull request? Fix IllegalReferenceCountException issues when exceptions happened in clientReadHandler.readShuffleData(). ### Why are the changes needed? A follow-up PR for: #1522 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing UTs.
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 24, 2024
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 24, 2024
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 24, 2024
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 24, 2024
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 24, 2024
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 24, 2024
jerqi
pushed a commit
that referenced
this issue
Feb 26, 2024
### What changes were proposed in this pull request? Fix [#1008](#1008). It does not actually test `GRPC_NETTY` mode, because it uses `ShuffleServerGrpcClient` everywhere instead of `ShuffleServerGrpcNettyClient`. Setting the shuffle server's tags to `GRPC_NETTY,GRPC` is useless, because we are not using `ShuffleServerGrpcNettyClient` at all. ### Why are the changes needed? It is a sub PR for: #1519 Also, it is a follow-up PR for: #1008 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing UTs.
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Feb 28, 2024
… shuffle data requests
rickyma
added a commit
to rickyma/incubator-uniffle
that referenced
this issue
Mar 21, 2024
…rately when failing to cache shuffle data
zuston
pushed a commit
that referenced
this issue
Mar 25, 2024
… when failing to cache shuffle data (#1597) ### What changes were proposed in this pull request? Release memory more accurately when failing to cache shuffle data. ### Why are the changes needed? A follow-up PR for: #1534. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing UTs.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Code of Conduct
Search before asking
Describe the bug
In high-pressure scenarios, inaccurate flow control(usedMemory? preAllocatedMemory?) leads to Shuffle server OOM.
The SQL used to reproduce the bug:
tpcds:
select * from (
select s.,c. from store_sales s join customer c on s.ss_customer_sk=c.c_customer_sk
) sc DISTRIBUTE BY sc.ss_customer_sk,sc.ss_item_sk;
Affects Version(s)
master
Uniffle Server Log Output
Uniffle Engine Log Output
No response
Uniffle Server Configurations
Uniffle Engine Configurations
Additional context
No response
Are you willing to submit PR?
The text was updated successfully, but these errors were encountered: