This repository has been archived by the owner on Sep 18, 2023. It is now read-only.
[NSE-516]Support ExternalSorter to control memory footage #517
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xuechendi
force-pushed
the
wip_externalsort
branch
4 times, most recently
from
15ef1c2
to
c12d400
Compare
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
… spill and read codes Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
xuechendi
force-pushed
the
wip_externalsort
branch
from
c12d400
to
d08ecda
Compare
1. spill during evaluate 2. spill during resultIter 3. need to add lock to avoid threading fail or inconsistency Signed-off-by: Chendi Xue <chendi.xue@intel.com>
xuechendi
changed the title
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
xuechendi
force-pushed
the
wip_externalsort
branch
from
bf4a835
to
317aed0
Compare
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Comment on lines
+2833
to
+2834
| RETURN_NOT_OK(impl_->Evaluate(in)); | ||
| RETURN_NOT_OK(impl_->MaybeSpill()); |
There was a problem hiding this comment.
Do this 2 lines acquire for memory from context memory pool too?
There was a problem hiding this comment.
MaybeSpill will check if cached data go beyond threshold, and then do spill and read the first batch from disk, so there maybe some allocation happened.
There was a problem hiding this comment.
So this may cause deadlock? Somewhere seemed hanging when I was testing the code. I made sure it was not a Spill->ReadBatch->Spill call. maybe it was MaybeSpill->ReadBatch->Spill
There was a problem hiding this comment.
maybespill is spill
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
2021/09/23
CPP part functionality test is done, all four cases are supported
Next