Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix incorrect usage of LinkedList in GroupByKeys #1530

Merged
merged 4 commits into from
Jul 23, 2020

Conversation

birdstorm
Copy link
Contributor

@birdstorm birdstorm commented Jul 21, 2020

No description provided.

@birdstorm birdstorm changed the title [DNM] Fix incorrect usage of LinkedList in GroupByKeys Fix incorrect usage of LinkedList in GroupByKeys Jul 23, 2020
@birdstorm birdstorm merged commit 2debc43 into pingcap:customer6 Jul 23, 2020
@birdstorm birdstorm deleted the customer6-optimize branch July 23, 2020 11:30
marsishandsome pushed a commit to marsishandsome/tispark that referenced this pull request Aug 12, 2020
marsishandsome added a commit that referenced this pull request Aug 19, 2020
* set enable region split default value to true

* fix txn heartbeat retry not invalidating region cache

* BatchWrite: add parameter taskNumPerRegion

* try to solve TTLManager TxnLockNotFound problem

* hack: use tispark to resolve locks

* Revert "try to solve TTLManager TxnLockNotFound problem"

This reverts commit 6c8d7c3.

* fix ordering null point exception

* Revert "hack: use tispark to resolve locks"

This reverts commit 6cc1eb5.

* 1. set txn_size in precommit request, 2. use sample to split index region

* fix oom - kvclient not closed after batch get

* fmt code

* fix NullPointerException: appendBatchBySize keys=null

* add parameter: spark.tispark.shuffleKeyToSameRegion, default=true

* add parameter: prewriteBackOfferMS default=240000

* add repartition

* fix escape char in jdbc url

* add retry for commit secondary keys

* add configuration to control whether to retry commit secondary keys

* fix TiRegionPartitioner if writeConcurrency is set

* check parameter taskNumPerRegion

* fix index split syntax

* fmt

* fix batch get resolve lock bug

* fix key not found bug (#1531)

* Fix inconsistent index in batch write (#1532)

* fix inconsistent index

* update test

* fix null unique index key encode error (#1529)

* fix null unique index key encode error

* fix bug

* fix bug

* fix bug

* fmt

* do not throw exception when split index failed

* Fix incorrect usage of LinkedList in GroupByKeys (#1530)

* fix split float/double index region (#1533)

* add check for region split when minVal = maxVal (#1537)

* use startTs's previous timestamp to read (#1536)

* fix toString in index split region

* add argument: txnCommitBatchSize & writeTaskNumber & writeBufferSize (#1538)

* fix resolve lock npe (#1539)

* fix resolve lock npe

* continue

* ignore WriteReadSuite test

* set snapshotBatchGetSize default value to 20480

* change TwoPhaseCommitter log level to info

* Revert "set snapshotBatchGetSize default value to 20480"

This reverts commit 2cd48ff.

* add invalidate region for batchGet

* add column name in error message (#1544)

* add argument: writeThreadPerTask (#1545)

* support commit concurrency (#1546)

* increase getRegionById backoffer

* fix Store Not Match error

* fix Store Not Match error in LockResolver

* Revert "fix Store Not Match error"

This reverts commit 1cfff82.

* refactor ThreadPool and parameter (#1548)

* fix BatchGet stuck bug (#1549)

* do not use getRegionById in retry logic (#1550)

* add more log for TwoPhaseCommitter (#1551)

* add argument commitBackOfferMS (#1552)

* continue run when meet exception during commit secondary key (#1553)

* Batch Write optimization (#1535)

* fmt

* delete unused PREWRITE_CONCURRENCY

* add variable: tidb write split region finish

* set commitBackOfferMS default value from 60s to 20s

* refactor tidb_wait_split_region_finish & fix sql exec bug

* add more log

* update writeSplitRegionFinish

* add more log

* Revert "continue run when meet exception during commit secondary key (#1553)"

This reverts commit 841adf7.

* region split version2 (#1558)

* add parameter: txnPrewriteBatchSize & txnCommitBatchSize (#1560)

* fix columnar batch (#1559)

* set TIDB_REGION_SPLIT_METHOD default to v2

Co-authored-by: xufei <xufeixw@mail.ustc.edu.cn>
Co-authored-by: birdstorm <samuelwyf@hotmail.com>
Co-authored-by: xufei <xufei@pingcap.com>
marsishandsome added a commit that referenced this pull request Aug 19, 2020
* set enable region split default value to true

* fix txn heartbeat retry not invalidating region cache

* BatchWrite: add parameter taskNumPerRegion

* try to solve TTLManager TxnLockNotFound problem

* hack: use tispark to resolve locks

* Revert "try to solve TTLManager TxnLockNotFound problem"

This reverts commit 6c8d7c3.

* fix ordering null point exception

* Revert "hack: use tispark to resolve locks"

This reverts commit 6cc1eb5.

* 1. set txn_size in precommit request, 2. use sample to split index region

* fix oom - kvclient not closed after batch get

* fmt code

* fix NullPointerException: appendBatchBySize keys=null

* add parameter: spark.tispark.shuffleKeyToSameRegion, default=true

* add parameter: prewriteBackOfferMS default=240000

* add repartition

* fix escape char in jdbc url

* add retry for commit secondary keys

* add configuration to control whether to retry commit secondary keys

* fix TiRegionPartitioner if writeConcurrency is set

* check parameter taskNumPerRegion

* fix index split syntax

* fmt

* fix batch get resolve lock bug

* fix key not found bug (#1531)

* Fix inconsistent index in batch write (#1532)

* fix inconsistent index

* update test

* fix null unique index key encode error (#1529)

* fix null unique index key encode error

* fix bug

* fix bug

* fix bug

* fmt

* do not throw exception when split index failed

* Fix incorrect usage of LinkedList in GroupByKeys (#1530)

* fix split float/double index region (#1533)

* add check for region split when minVal = maxVal (#1537)

* use startTs's previous timestamp to read (#1536)

* fix toString in index split region

* add argument: txnCommitBatchSize & writeTaskNumber & writeBufferSize (#1538)

* fix resolve lock npe (#1539)

* fix resolve lock npe

* continue

* ignore WriteReadSuite test

* set snapshotBatchGetSize default value to 20480

* change TwoPhaseCommitter log level to info

* Revert "set snapshotBatchGetSize default value to 20480"

This reverts commit 2cd48ff.

* add invalidate region for batchGet

* add column name in error message (#1544)

* add argument: writeThreadPerTask (#1545)

* support commit concurrency (#1546)

* increase getRegionById backoffer

* fix Store Not Match error

* fix Store Not Match error in LockResolver

* Revert "fix Store Not Match error"

This reverts commit 1cfff82.

* refactor ThreadPool and parameter (#1548)

* fix BatchGet stuck bug (#1549)

* do not use getRegionById in retry logic (#1550)

* add more log for TwoPhaseCommitter (#1551)

* add argument commitBackOfferMS (#1552)

* continue run when meet exception during commit secondary key (#1553)

* Batch Write optimization (#1535)

* fmt

* delete unused PREWRITE_CONCURRENCY

* add variable: tidb write split region finish

* set commitBackOfferMS default value from 60s to 20s

* refactor tidb_wait_split_region_finish & fix sql exec bug

* add more log

* update writeSplitRegionFinish

* add more log

* Revert "continue run when meet exception during commit secondary key (#1553)"

This reverts commit 841adf7.

* region split version2 (#1558)

* add parameter: txnPrewriteBatchSize & txnCommitBatchSize (#1560)

* fix columnar batch (#1559)

* set TIDB_REGION_SPLIT_METHOD default to v2

Co-authored-by: xufei <xufeixw@mail.ustc.edu.cn>
Co-authored-by: birdstorm <samuelwyf@hotmail.com>
Co-authored-by: xufei <xufei@pingcap.com>
wfxxh pushed a commit to wanfangdata/tispark that referenced this pull request Jun 30, 2023
* set enable region split default value to true

* fix txn heartbeat retry not invalidating region cache

* BatchWrite: add parameter taskNumPerRegion

* try to solve TTLManager TxnLockNotFound problem

* hack: use tispark to resolve locks

* Revert "try to solve TTLManager TxnLockNotFound problem"

This reverts commit 6c8d7c3.

* fix ordering null point exception

* Revert "hack: use tispark to resolve locks"

This reverts commit 6cc1eb5.

* 1. set txn_size in precommit request, 2. use sample to split index region

* fix oom - kvclient not closed after batch get

* fmt code

* fix NullPointerException: appendBatchBySize keys=null

* add parameter: spark.tispark.shuffleKeyToSameRegion, default=true

* add parameter: prewriteBackOfferMS default=240000

* add repartition

* fix escape char in jdbc url

* add retry for commit secondary keys

* add configuration to control whether to retry commit secondary keys

* fix TiRegionPartitioner if writeConcurrency is set

* check parameter taskNumPerRegion

* fix index split syntax

* fmt

* fix batch get resolve lock bug

* fix key not found bug (pingcap#1531)

* Fix inconsistent index in batch write (pingcap#1532)

* fix inconsistent index

* update test

* fix null unique index key encode error (pingcap#1529)

* fix null unique index key encode error

* fix bug

* fix bug

* fix bug

* fmt

* do not throw exception when split index failed

* Fix incorrect usage of LinkedList in GroupByKeys (pingcap#1530)

* fix split float/double index region (pingcap#1533)

* add check for region split when minVal = maxVal (pingcap#1537)

* use startTs's previous timestamp to read (pingcap#1536)

* fix toString in index split region

* add argument: txnCommitBatchSize & writeTaskNumber & writeBufferSize (pingcap#1538)

* fix resolve lock npe (pingcap#1539)

* fix resolve lock npe

* continue

* ignore WriteReadSuite test

* set snapshotBatchGetSize default value to 20480

* change TwoPhaseCommitter log level to info

* Revert "set snapshotBatchGetSize default value to 20480"

This reverts commit 2cd48ff.

* add invalidate region for batchGet

* add column name in error message (pingcap#1544)

* add argument: writeThreadPerTask (pingcap#1545)

* support commit concurrency (pingcap#1546)

* increase getRegionById backoffer

* fix Store Not Match error

* fix Store Not Match error in LockResolver

* Revert "fix Store Not Match error"

This reverts commit 1cfff82.

* refactor ThreadPool and parameter (pingcap#1548)

* fix BatchGet stuck bug (pingcap#1549)

* do not use getRegionById in retry logic (pingcap#1550)

* add more log for TwoPhaseCommitter (pingcap#1551)

* add argument commitBackOfferMS (pingcap#1552)

* continue run when meet exception during commit secondary key (pingcap#1553)

* Batch Write optimization (pingcap#1535)

* fmt

* delete unused PREWRITE_CONCURRENCY

* add variable: tidb write split region finish

* set commitBackOfferMS default value from 60s to 20s

* refactor tidb_wait_split_region_finish & fix sql exec bug

* add more log

* update writeSplitRegionFinish

* add more log

* Revert "continue run when meet exception during commit secondary key (pingcap#1553)"

This reverts commit 841adf7.

* region split version2 (pingcap#1558)

* add parameter: txnPrewriteBatchSize & txnCommitBatchSize (pingcap#1560)

* fix columnar batch (pingcap#1559)

* set TIDB_REGION_SPLIT_METHOD default to v2

Co-authored-by: xufei <xufeixw@mail.ustc.edu.cn>
Co-authored-by: birdstorm <samuelwyf@hotmail.com>
Co-authored-by: xufei <xufei@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant