This repository has been archived by the owner on Sep 18, 2023. It is now read-only.
[NSE-928] Add ARROW_CHECK for batch_size check #973
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FelixYBW
reviewed
Jun 15, 2022
| @@ -854,7 +854,7 @@ Splitter::row_offset_type Splitter::CalculateSplitBatchSize( | |||
|
|
|||
| arrow::Status Splitter::DoSplit(const arrow::RecordBatch& rb) { | |||
| // buffer is allocated less than 64K | |||
| // ARROW_CHECK_LE(rb.num_rows(),64*1024); | |||
| ARROW_CHECK_LE(rb.num_rows(), 64 * 1024); | |||
There was a problem hiding this comment.
It fails in TPCDS power test if we add the check here. We need to fix the max batch issue before we enable the check here
| @@ -1399,7 +1399,9 @@ class SortOnekeyKernel : public SortArraysToIndicesKernel::Impl { | |||
| if (nulls_total_ == 0) { | |||
| // if all batches have no null value, | |||
| // we do not need to check whether the value is null | |||
| ARROW_CHECK_LE(num_batches_, 64 * 1024); | |||
There was a problem hiding this comment.
num_batches_ is controled by partition size eventually. looks limitation of 64K maximal recordbatch is too small.
|
If use uint32_t for ArrayItemIndex: array_id, caused many problems: crash and result mismatch. It is noted in #989 . So retain uint16_t ，and troubleshoot later. |
|
cc @zhouyuan |
|
Passed Jenkins workloads again. |
zhouyuan
added a commit
to zhouyuan/native-sql-engine
that referenced
this pull request
Jul 27, 2022
…)" This reverts commit c4d4a65.
zhouyuan
added a commit
that referenced
this pull request
Aug 10, 2022
* remove sort limit Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * s/uint16/uint32 Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * Revert "s/uint16/uint32" This reverts commit 35c0909. * Revert "remove sort limit" This reverts commit 6b5738a. * change sort limit Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * Revert "[NSE-928] Add ARROW_CHECK for batch_size check (#973)" This reverts commit c4d4a65. * fix window sort Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)