fix batch write bug #1562
Merged
fix batch write bug #1562
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ingcap#1553)" This reverts commit 841adf7.
marsishandsome
added
the
needs-cherry-pick-master
|
/run-all-tests tikv=v3.0.14 tidb=v3.0.14 pd=v3.0.14 |
marsishandsome
force-pushed
the
feature/batch-write-customer-fix
branch
from
1ef8337 to
a2f2673
Compare
|
/run-all-tests tikv=v3.0.14 tidb=v3.0.14 pd=v3.0.14 |
…write-customer-fix
marsishandsome
force-pushed
the
feature/batch-write-customer-fix
branch
from
a2f2673 to
a48a5ce
Compare
|
/run-all-tests tikv=v3.0.14 tidb=v3.0.14 pd=v3.0.14 |
|
cherry pick to master failed |
marsishandsome
added a commit
that referenced
this pull request
Aug 19, 2020
* set enable region split default value to true * fix txn heartbeat retry not invalidating region cache * BatchWrite: add parameter taskNumPerRegion * try to solve TTLManager TxnLockNotFound problem * hack: use tispark to resolve locks * Revert "try to solve TTLManager TxnLockNotFound problem" This reverts commit 6c8d7c3. * fix ordering null point exception * Revert "hack: use tispark to resolve locks" This reverts commit 6cc1eb5. * 1. set txn_size in precommit request, 2. use sample to split index region * fix oom - kvclient not closed after batch get * fmt code * fix NullPointerException: appendBatchBySize keys=null * add parameter: spark.tispark.shuffleKeyToSameRegion, default=true * add parameter: prewriteBackOfferMS default=240000 * add repartition * fix escape char in jdbc url * add retry for commit secondary keys * add configuration to control whether to retry commit secondary keys * fix TiRegionPartitioner if writeConcurrency is set * check parameter taskNumPerRegion * fix index split syntax * fmt * fix batch get resolve lock bug * fix key not found bug (#1531) * Fix inconsistent index in batch write (#1532) * fix inconsistent index * update test * fix null unique index key encode error (#1529) * fix null unique index key encode error * fix bug * fix bug * fix bug * fmt * do not throw exception when split index failed * Fix incorrect usage of LinkedList in GroupByKeys (#1530) * fix split float/double index region (#1533) * add check for region split when minVal = maxVal (#1537) * use startTs's previous timestamp to read (#1536) * fix toString in index split region * add argument: txnCommitBatchSize & writeTaskNumber & writeBufferSize (#1538) * fix resolve lock npe (#1539) * fix resolve lock npe * continue * ignore WriteReadSuite test * set snapshotBatchGetSize default value to 20480 * change TwoPhaseCommitter log level to info * Revert "set snapshotBatchGetSize default value to 20480" This reverts commit 2cd48ff. * add invalidate region for batchGet * add column name in error message (#1544) * add argument: writeThreadPerTask (#1545) * support commit concurrency (#1546) * increase getRegionById backoffer * fix Store Not Match error * fix Store Not Match error in LockResolver * Revert "fix Store Not Match error" This reverts commit 1cfff82. * refactor ThreadPool and parameter (#1548) * fix BatchGet stuck bug (#1549) * do not use getRegionById in retry logic (#1550) * add more log for TwoPhaseCommitter (#1551) * add argument commitBackOfferMS (#1552) * continue run when meet exception during commit secondary key (#1553) * Batch Write optimization (#1535) * fmt * delete unused PREWRITE_CONCURRENCY * add variable: tidb write split region finish * set commitBackOfferMS default value from 60s to 20s * refactor tidb_wait_split_region_finish & fix sql exec bug * add more log * update writeSplitRegionFinish * add more log * Revert "continue run when meet exception during commit secondary key (#1553)" This reverts commit 841adf7. * region split version2 (#1558) * add parameter: txnPrewriteBatchSize & txnCommitBatchSize (#1560) * fix columnar batch (#1559) * set TIDB_REGION_SPLIT_METHOD default to v2 Co-authored-by: xufei <xufeixw@mail.ustc.edu.cn> Co-authored-by: birdstorm <samuelwyf@hotmail.com> Co-authored-by: xufei <xufei@pingcap.com>
wfxxh
pushed a commit
to wanfangdata/tispark
that referenced
this pull request
Jun 30, 2023
* set enable region split default value to true * fix txn heartbeat retry not invalidating region cache * BatchWrite: add parameter taskNumPerRegion * try to solve TTLManager TxnLockNotFound problem * hack: use tispark to resolve locks * Revert "try to solve TTLManager TxnLockNotFound problem" This reverts commit 6c8d7c3. * fix ordering null point exception * Revert "hack: use tispark to resolve locks" This reverts commit 6cc1eb5. * 1. set txn_size in precommit request, 2. use sample to split index region * fix oom - kvclient not closed after batch get * fmt code * fix NullPointerException: appendBatchBySize keys=null * add parameter: spark.tispark.shuffleKeyToSameRegion, default=true * add parameter: prewriteBackOfferMS default=240000 * add repartition * fix escape char in jdbc url * add retry for commit secondary keys * add configuration to control whether to retry commit secondary keys * fix TiRegionPartitioner if writeConcurrency is set * check parameter taskNumPerRegion * fix index split syntax * fmt * fix batch get resolve lock bug * fix key not found bug (pingcap#1531) * Fix inconsistent index in batch write (pingcap#1532) * fix inconsistent index * update test * fix null unique index key encode error (pingcap#1529) * fix null unique index key encode error * fix bug * fix bug * fix bug * fmt * do not throw exception when split index failed * Fix incorrect usage of LinkedList in GroupByKeys (pingcap#1530) * fix split float/double index region (pingcap#1533) * add check for region split when minVal = maxVal (pingcap#1537) * use startTs's previous timestamp to read (pingcap#1536) * fix toString in index split region * add argument: txnCommitBatchSize & writeTaskNumber & writeBufferSize (pingcap#1538) * fix resolve lock npe (pingcap#1539) * fix resolve lock npe * continue * ignore WriteReadSuite test * set snapshotBatchGetSize default value to 20480 * change TwoPhaseCommitter log level to info * Revert "set snapshotBatchGetSize default value to 20480" This reverts commit 2cd48ff. * add invalidate region for batchGet * add column name in error message (pingcap#1544) * add argument: writeThreadPerTask (pingcap#1545) * support commit concurrency (pingcap#1546) * increase getRegionById backoffer * fix Store Not Match error * fix Store Not Match error in LockResolver * Revert "fix Store Not Match error" This reverts commit 1cfff82. * refactor ThreadPool and parameter (pingcap#1548) * fix BatchGet stuck bug (pingcap#1549) * do not use getRegionById in retry logic (pingcap#1550) * add more log for TwoPhaseCommitter (pingcap#1551) * add argument commitBackOfferMS (pingcap#1552) * continue run when meet exception during commit secondary key (pingcap#1553) * Batch Write optimization (pingcap#1535) * fmt * delete unused PREWRITE_CONCURRENCY * add variable: tidb write split region finish * set commitBackOfferMS default value from 60s to 20s * refactor tidb_wait_split_region_finish & fix sql exec bug * add more log * update writeSplitRegionFinish * add more log * Revert "continue run when meet exception during commit secondary key (pingcap#1553)" This reverts commit 841adf7. * region split version2 (pingcap#1558) * add parameter: txnPrewriteBatchSize & txnCommitBatchSize (pingcap#1560) * fix columnar batch (pingcap#1559) * set TIDB_REGION_SPLIT_METHOD default to v2 Co-authored-by: xufei <xufeixw@mail.ustc.edu.cn> Co-authored-by: birdstorm <samuelwyf@hotmail.com> Co-authored-by: xufei <xufei@pingcap.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is changed and how it works?
fix a lot of batch write bugs