Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix columnar batch #1559

Merged
merged 6 commits into from
Aug 19, 2020
Merged

Conversation

birdstorm
Copy link
Contributor

What problem does this PR solve?

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Code changes

  • Has exported function/method change
  • Has exported variable/fields change
  • Has interface methods change
  • Has persistent data change

Side effects

  • Possible performance regression
  • Increased code complexity
  • Breaking backward compatibility

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation
  • Need to update the tidb-ansible repository
  • Need to be included in the release note

@birdstorm
Copy link
Contributor Author

/run-all-tests pd=v3.0.14 tikv=v3.0.14 tidb=v3.0.14

@birdstorm
Copy link
Contributor Author

/run-all-tests pd=v3.0.14 tikv=v3.0.14 tidb=v3.0.14

@birdstorm
Copy link
Contributor Author

/run-all-tests pd=v3.0.14 tikv=v3.0.14 tidb=v3.0.14

@birdstorm
Copy link
Contributor Author

/run-all-tests pd=v3.0.14 tikv=v3.0.14 tidb=v3.0.14

Copy link
Collaborator

@marsishandsome marsishandsome left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@marsishandsome
Copy link
Collaborator

/run-all-tests pd=v3.0.14 tikv=v3.0.14 tidb=v3.0.14

@marsishandsome marsishandsome merged commit 5451b76 into pingcap:customer6 Aug 19, 2020
@birdstorm birdstorm deleted the fix-columnar-batch branch August 19, 2020 07:16
marsishandsome added a commit that referenced this pull request Aug 19, 2020
* set enable region split default value to true

* fix txn heartbeat retry not invalidating region cache

* BatchWrite: add parameter taskNumPerRegion

* try to solve TTLManager TxnLockNotFound problem

* hack: use tispark to resolve locks

* Revert "try to solve TTLManager TxnLockNotFound problem"

This reverts commit 6c8d7c3.

* fix ordering null point exception

* Revert "hack: use tispark to resolve locks"

This reverts commit 6cc1eb5.

* 1. set txn_size in precommit request, 2. use sample to split index region

* fix oom - kvclient not closed after batch get

* fmt code

* fix NullPointerException: appendBatchBySize keys=null

* add parameter: spark.tispark.shuffleKeyToSameRegion, default=true

* add parameter: prewriteBackOfferMS default=240000

* add repartition

* fix escape char in jdbc url

* add retry for commit secondary keys

* add configuration to control whether to retry commit secondary keys

* fix TiRegionPartitioner if writeConcurrency is set

* check parameter taskNumPerRegion

* fix index split syntax

* fmt

* fix batch get resolve lock bug

* fix key not found bug (#1531)

* Fix inconsistent index in batch write (#1532)

* fix inconsistent index

* update test

* fix null unique index key encode error (#1529)

* fix null unique index key encode error

* fix bug

* fix bug

* fix bug

* fmt

* do not throw exception when split index failed

* Fix incorrect usage of LinkedList in GroupByKeys (#1530)

* fix split float/double index region (#1533)

* add check for region split when minVal = maxVal (#1537)

* use startTs's previous timestamp to read (#1536)

* fix toString in index split region

* add argument: txnCommitBatchSize & writeTaskNumber & writeBufferSize (#1538)

* fix resolve lock npe (#1539)

* fix resolve lock npe

* continue

* ignore WriteReadSuite test

* set snapshotBatchGetSize default value to 20480

* change TwoPhaseCommitter log level to info

* Revert "set snapshotBatchGetSize default value to 20480"

This reverts commit 2cd48ff.

* add invalidate region for batchGet

* add column name in error message (#1544)

* add argument: writeThreadPerTask (#1545)

* support commit concurrency (#1546)

* increase getRegionById backoffer

* fix Store Not Match error

* fix Store Not Match error in LockResolver

* Revert "fix Store Not Match error"

This reverts commit 1cfff82.

* refactor ThreadPool and parameter (#1548)

* fix BatchGet stuck bug (#1549)

* do not use getRegionById in retry logic (#1550)

* add more log for TwoPhaseCommitter (#1551)

* add argument commitBackOfferMS (#1552)

* continue run when meet exception during commit secondary key (#1553)

* Batch Write optimization (#1535)

* fmt

* delete unused PREWRITE_CONCURRENCY

* add variable: tidb write split region finish

* set commitBackOfferMS default value from 60s to 20s

* refactor tidb_wait_split_region_finish & fix sql exec bug

* add more log

* update writeSplitRegionFinish

* add more log

* Revert "continue run when meet exception during commit secondary key (#1553)"

This reverts commit 841adf7.

* region split version2 (#1558)

* add parameter: txnPrewriteBatchSize & txnCommitBatchSize (#1560)

* fix columnar batch (#1559)

* set TIDB_REGION_SPLIT_METHOD default to v2

Co-authored-by: xufei <xufeixw@mail.ustc.edu.cn>
Co-authored-by: birdstorm <samuelwyf@hotmail.com>
Co-authored-by: xufei <xufei@pingcap.com>
marsishandsome added a commit that referenced this pull request Aug 19, 2020
* set enable region split default value to true

* fix txn heartbeat retry not invalidating region cache

* BatchWrite: add parameter taskNumPerRegion

* try to solve TTLManager TxnLockNotFound problem

* hack: use tispark to resolve locks

* Revert "try to solve TTLManager TxnLockNotFound problem"

This reverts commit 6c8d7c3.

* fix ordering null point exception

* Revert "hack: use tispark to resolve locks"

This reverts commit 6cc1eb5.

* 1. set txn_size in precommit request, 2. use sample to split index region

* fix oom - kvclient not closed after batch get

* fmt code

* fix NullPointerException: appendBatchBySize keys=null

* add parameter: spark.tispark.shuffleKeyToSameRegion, default=true

* add parameter: prewriteBackOfferMS default=240000

* add repartition

* fix escape char in jdbc url

* add retry for commit secondary keys

* add configuration to control whether to retry commit secondary keys

* fix TiRegionPartitioner if writeConcurrency is set

* check parameter taskNumPerRegion

* fix index split syntax

* fmt

* fix batch get resolve lock bug

* fix key not found bug (#1531)

* Fix inconsistent index in batch write (#1532)

* fix inconsistent index

* update test

* fix null unique index key encode error (#1529)

* fix null unique index key encode error

* fix bug

* fix bug

* fix bug

* fmt

* do not throw exception when split index failed

* Fix incorrect usage of LinkedList in GroupByKeys (#1530)

* fix split float/double index region (#1533)

* add check for region split when minVal = maxVal (#1537)

* use startTs's previous timestamp to read (#1536)

* fix toString in index split region

* add argument: txnCommitBatchSize & writeTaskNumber & writeBufferSize (#1538)

* fix resolve lock npe (#1539)

* fix resolve lock npe

* continue

* ignore WriteReadSuite test

* set snapshotBatchGetSize default value to 20480

* change TwoPhaseCommitter log level to info

* Revert "set snapshotBatchGetSize default value to 20480"

This reverts commit 2cd48ff.

* add invalidate region for batchGet

* add column name in error message (#1544)

* add argument: writeThreadPerTask (#1545)

* support commit concurrency (#1546)

* increase getRegionById backoffer

* fix Store Not Match error

* fix Store Not Match error in LockResolver

* Revert "fix Store Not Match error"

This reverts commit 1cfff82.

* refactor ThreadPool and parameter (#1548)

* fix BatchGet stuck bug (#1549)

* do not use getRegionById in retry logic (#1550)

* add more log for TwoPhaseCommitter (#1551)

* add argument commitBackOfferMS (#1552)

* continue run when meet exception during commit secondary key (#1553)

* Batch Write optimization (#1535)

* fmt

* delete unused PREWRITE_CONCURRENCY

* add variable: tidb write split region finish

* set commitBackOfferMS default value from 60s to 20s

* refactor tidb_wait_split_region_finish & fix sql exec bug

* add more log

* update writeSplitRegionFinish

* add more log

* Revert "continue run when meet exception during commit secondary key (#1553)"

This reverts commit 841adf7.

* region split version2 (#1558)

* add parameter: txnPrewriteBatchSize & txnCommitBatchSize (#1560)

* fix columnar batch (#1559)

* set TIDB_REGION_SPLIT_METHOD default to v2

Co-authored-by: xufei <xufeixw@mail.ustc.edu.cn>
Co-authored-by: birdstorm <samuelwyf@hotmail.com>
Co-authored-by: xufei <xufei@pingcap.com>
marsishandsome added a commit to marsishandsome/tispark that referenced this pull request Aug 19, 2020
wfxxh pushed a commit to wanfangdata/tispark that referenced this pull request Jun 30, 2023
* set enable region split default value to true

* fix txn heartbeat retry not invalidating region cache

* BatchWrite: add parameter taskNumPerRegion

* try to solve TTLManager TxnLockNotFound problem

* hack: use tispark to resolve locks

* Revert "try to solve TTLManager TxnLockNotFound problem"

This reverts commit 6c8d7c3.

* fix ordering null point exception

* Revert "hack: use tispark to resolve locks"

This reverts commit 6cc1eb5.

* 1. set txn_size in precommit request, 2. use sample to split index region

* fix oom - kvclient not closed after batch get

* fmt code

* fix NullPointerException: appendBatchBySize keys=null

* add parameter: spark.tispark.shuffleKeyToSameRegion, default=true

* add parameter: prewriteBackOfferMS default=240000

* add repartition

* fix escape char in jdbc url

* add retry for commit secondary keys

* add configuration to control whether to retry commit secondary keys

* fix TiRegionPartitioner if writeConcurrency is set

* check parameter taskNumPerRegion

* fix index split syntax

* fmt

* fix batch get resolve lock bug

* fix key not found bug (pingcap#1531)

* Fix inconsistent index in batch write (pingcap#1532)

* fix inconsistent index

* update test

* fix null unique index key encode error (pingcap#1529)

* fix null unique index key encode error

* fix bug

* fix bug

* fix bug

* fmt

* do not throw exception when split index failed

* Fix incorrect usage of LinkedList in GroupByKeys (pingcap#1530)

* fix split float/double index region (pingcap#1533)

* add check for region split when minVal = maxVal (pingcap#1537)

* use startTs's previous timestamp to read (pingcap#1536)

* fix toString in index split region

* add argument: txnCommitBatchSize & writeTaskNumber & writeBufferSize (pingcap#1538)

* fix resolve lock npe (pingcap#1539)

* fix resolve lock npe

* continue

* ignore WriteReadSuite test

* set snapshotBatchGetSize default value to 20480

* change TwoPhaseCommitter log level to info

* Revert "set snapshotBatchGetSize default value to 20480"

This reverts commit 2cd48ff.

* add invalidate region for batchGet

* add column name in error message (pingcap#1544)

* add argument: writeThreadPerTask (pingcap#1545)

* support commit concurrency (pingcap#1546)

* increase getRegionById backoffer

* fix Store Not Match error

* fix Store Not Match error in LockResolver

* Revert "fix Store Not Match error"

This reverts commit 1cfff82.

* refactor ThreadPool and parameter (pingcap#1548)

* fix BatchGet stuck bug (pingcap#1549)

* do not use getRegionById in retry logic (pingcap#1550)

* add more log for TwoPhaseCommitter (pingcap#1551)

* add argument commitBackOfferMS (pingcap#1552)

* continue run when meet exception during commit secondary key (pingcap#1553)

* Batch Write optimization (pingcap#1535)

* fmt

* delete unused PREWRITE_CONCURRENCY

* add variable: tidb write split region finish

* set commitBackOfferMS default value from 60s to 20s

* refactor tidb_wait_split_region_finish & fix sql exec bug

* add more log

* update writeSplitRegionFinish

* add more log

* Revert "continue run when meet exception during commit secondary key (pingcap#1553)"

This reverts commit 841adf7.

* region split version2 (pingcap#1558)

* add parameter: txnPrewriteBatchSize & txnCommitBatchSize (pingcap#1560)

* fix columnar batch (pingcap#1559)

* set TIDB_REGION_SPLIT_METHOD default to v2

Co-authored-by: xufei <xufeixw@mail.ustc.edu.cn>
Co-authored-by: birdstorm <samuelwyf@hotmail.com>
Co-authored-by: xufei <xufei@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants