Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

br: batch ddl for binlog #31028

Merged
merged 577 commits into from
Dec 26, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
577 commits
Select commit Hold shift + click to select a range
805a887
executor: migrate test-infra to testify for executor/show_stats_test.…
hawkingrei Nov 23, 2021
2f119e3
ddl/table_split_test.go: migrate test-infra to testify (#29960)
wangggong Nov 23, 2021
e030ff1
*: fix unstable test caused by TestRandomPanicAggConsume (#30035)
lcwangchao Nov 23, 2021
efd6c5e
ddl/ddl_test.go: refactor testSchemaInfo to return error #29964 (#30008)
clovis1122 Nov 23, 2021
759aab5
executor: remove useless parameter (#30043)
hawkingrei Nov 23, 2021
e453873
executor: fix error msg of granting non-table level privilege (#29321)
unconsolable Nov 23, 2021
0f86a53
server: fix bug https://asktug.com/t/topic/213082/11 (#29577)
zgcbj Nov 23, 2021
fb5eb1f
*: replace compareDatum by compare (#30048)
wjhuang2016 Nov 23, 2021
205d9ab
executor, store: replace compareDatum by compare (#30044)
wjhuang2016 Nov 23, 2021
bba5fee
executor: migrate test-infra to testify for executor/write_test.go (#…
hawkingrei Nov 23, 2021
7fdafb4
planner: add sub plan info of shuffleReceiver when query explain anal…
mmyj Nov 23, 2021
8858592
expression: fix wrong flen for CastAsString funtion (#29563)
wshwsh12 Nov 23, 2021
cc8617e
br: migrate test-infra to testify for redact (#29747)
tisonkun Nov 23, 2021
d252fd1
ddl: stable create expression index test (#29990)
wjhuang2016 Nov 23, 2021
8b028b8
*: replace compareDatum by compare (#30060)
wjhuang2016 Nov 23, 2021
18381f9
planner: fix the issue that binding cannot work when sql_select_limit…
qw4990 Nov 23, 2021
f7ff198
br : migrate test-infra to testify for lightning/backend/tidb/tidb_te…
JayLZhou Nov 23, 2021
9535f05
*: fix unstable test in placement gc cases (#30045)
lcwangchao Nov 23, 2021
c687915
lightning: let ignore columns be compatible with tidb backend (#27850)
glorv Nov 23, 2021
4051cf4
statistics: replace compareDatum by compare (#30052)
wjhuang2016 Nov 23, 2021
71e64bf
br/pkg/mock: migrate test-infra to testify (#30034)
tisonkun Nov 23, 2021
b57c1e1
ddl: forbit alter table cache in system db (#29998)
sylzd Nov 24, 2021
b11171a
topsql: distinguish the row and index operation type (#29044)
mornyx Nov 24, 2021
d04a128
ddl, util/codec: fix add index OOM and prevent panic in logging (#29925)
tangenta Nov 24, 2021
791f59d
br/pkg/utils: migrate tests to testify (#30032)
sleepymole Nov 24, 2021
45836a6
expression, sessionctx: support rand_seed1 and rand_seed2 sysvar (#29…
Defined2014 Nov 24, 2021
4814557
*: track the memory usage of IndexJoin more accurate (#29068)
XuHuaiyu Nov 24, 2021
f45a8f0
br : migrate test-infra to testify for lightning/backend/importer (#3…
JayLZhou Nov 24, 2021
05ccb8a
planner: fix panic when the join key is scalarFunction (#30002)
Yisaer Nov 24, 2021
a90c988
ddl: fix none info of `alter table placement` (#29929)
sylzd Nov 24, 2021
3d26751
expression: cast charset according to the function's resulting charse…
xiongjiwei Nov 24, 2021
b5ded48
*: replace compareDatum by compare and fix compare (#30090)
wjhuang2016 Nov 24, 2021
5916672
store/driver: Use BatchGet method supported in client-go. (#29860)
dwangxxx Nov 24, 2021
e65f548
privilege: disable role privilege when it is revoked from an user (#3…
lcwangchao Nov 24, 2021
9cd0b49
partition: Show partition reformat (#29945)
mjonss Nov 24, 2021
a75143d
planner: fix the unstable test `TestPartitionWithVariedDatasources` (…
qw4990 Nov 25, 2021
a8ca637
server: Add a `MockConn` to enable testing connections (#30119)
djshow832 Nov 25, 2021
dde1e0b
*: skip unstable test (#29433)
Yisaer Nov 25, 2021
7f93b09
table/tables: add `StateRemote` interface for the cached table (#29152)
tiancaiamao Nov 25, 2021
78c653e
planner: rebuild range when the range is empty (#30003)
Reminiscent Nov 25, 2021
fe5733c
planner: support dump file for trace plan statement (#30059)
Yisaer Nov 25, 2021
8dc59e6
planner: make clear for MaybeOverOptimized4PlanCache (#29782)
Reminiscent Nov 25, 2021
bee016b
update: fix the updatable table name resolution in build update list …
AilinKid Nov 25, 2021
e15b768
expression : prevent function DATE_ADD/SUB_STRING_XXX pushed down to …
mengxin9014 Nov 25, 2021
d6e9986
*: remove unused profile memory tracker from global tracker (#30143)
Yisaer Nov 25, 2021
75f6225
expression, util/codec: fix wrongly eliminated conditions caused by `…
time-and-fate Nov 25, 2021
bccc21b
cmd, expression: fix convert binary string to gbk (#30004)
Defined2014 Nov 26, 2021
f5af99e
sessionctx/variable: make lc_time_names read only (#30084)
espresso98 Nov 26, 2021
51b7a19
expression: fix misuse of datumsToConstants in GBK test (#30072)
unconsolable Nov 26, 2021
a9f161d
plugin: add more tests to cover audit logs (#30165)
djshow832 Nov 26, 2021
126dbc8
planner: consider prefix index column length in skyline pruning (#27527)
xuyifangreeneyes Nov 26, 2021
7a8c7f8
server: support download optimize trace file (#30150)
Yisaer Nov 26, 2021
3dc09f8
expression: add ut config (#30156)
hawkingrei Nov 26, 2021
4f19445
ddl/ddl_test.go: refactor testTableInfo to return error (#30069)
clovis1122 Nov 26, 2021
306621e
*: skip a case in TestClusterTables/ForClusterServerInfo on mac M1 (#…
zhangyangyu Nov 28, 2021
6d7a320
planner: do not add extra limit when handle the execute stmt (#30152)
Reminiscent Nov 29, 2021
437d8ba
ddl: migrate test-infra to testify for db_cache_test.go (#30117)
hawkingrei Nov 29, 2021
724da32
tables: fix data race in mockStatRemoteData (#30180)
hawkingrei Nov 29, 2021
11974ec
server: Port missing in processlist (#30183)
mjonss Nov 29, 2021
10d390d
server: add build not race flag (#30184)
hawkingrei Nov 29, 2021
df113a1
unistore: get/batchGet/scan support read-through-lock (#29898)
youjiali1995 Nov 29, 2021
4cf7051
topsql: fix nil pointer panic in stmtctx (#30181)
mornyx Nov 29, 2021
352811d
types: fix wrong str_to_date() microseconds with leading zeros are no…
wjhuang2016 Nov 29, 2021
4b826a0
dumpling: Add support for `Create Placement` (#29724)
sylzd Nov 29, 2021
8550dbb
*: add cardinality estimation trace for `Selectivity` (#29883)
time-and-fate Nov 29, 2021
eb7672f
*: replace compareDatum by compare and ignore warnings is collate is …
wjhuang2016 Nov 29, 2021
3baa95f
planner: implement aggregation eliminate optimize trace (#30114)
Yisaer Nov 29, 2021
6756bd2
server: Combined fix for authentication issues (#29738)
dveeden Nov 29, 2021
3487bb7
executer: fix data race (#30186)
hawkingrei Nov 29, 2021
87f4c7d
dumpling : add a function for the variable call of dm (#30033)
docsir Nov 29, 2021
8ace8f6
planner: revise optimize trace logic (#30163)
Yisaer Nov 29, 2021
5c8188e
statistics: fix unstable test case TestTraceCE (#30235)
time-and-fate Nov 29, 2021
1b6ffe9
lightning: avoid retry write rows and record duplicate rows in tidb b…
glorv Nov 29, 2021
443f15e
executor: send a task with error to the resultCh when panic happen (#…
XuHuaiyu Nov 29, 2021
6efa36d
docs: update placement policy limits (#30202)
morgo Nov 29, 2021
b1ca2eb
expression: Support GBK for builtin function AesEncrypt. (#29946)
jackwener Nov 29, 2021
81ab3f6
executor: remove useless log (#30248)
XuHuaiyu Nov 30, 2021
8fa0cae
docs: Update 2021-09-29-secure-bootstrap.md (#30243)
JameyWoo Nov 30, 2021
0c8614a
executor: fix unstable test of topsql (#30257)
mornyx Nov 30, 2021
37e0dac
copr: add paging API for streaming-like process (#29612)
you06 Nov 30, 2021
a0f7643
*: forbid set TiFlash Replica for a table with unsupport charset (#30…
Defined2014 Nov 30, 2021
7fc6ebb
privilege, session, server: consistently map user login to identity (…
morgo Nov 30, 2021
083be6d
expression: fix flen for timestamp_add (#30213)
wjhuang2016 Nov 30, 2021
0b2d9d1
types: check values equals to NaN or Inf when convert to float (#30148)
Defined2014 Nov 30, 2021
13d0deb
parser: respect TiDB comment when DROP INDEX IF EXISTS (#30173)
lance6716 Nov 30, 2021
13b7a2d
ddl: disallow change columns from zero int to time (#25728)
tangenta Nov 30, 2021
426a527
* : statement summary should know the slow write is blocked on read l…
JayLZhou Dec 1, 2021
a90344c
planner: regard NULL as point when accessing composite index (#30244)
qw4990 Dec 1, 2021
a046014
planner: Add trace for proj elimination rule (#30275)
Yisaer Dec 1, 2021
7c5e854
privilege,session: Match loopback connections to 'localhost' accounts…
mjonss Dec 1, 2021
e97e8c0
ddltest: refactor logutil.InitLogger in ddltest to avoid data race (#…
hawkingrei Dec 1, 2021
80d18b6
load data: fix bug if load data with long content (#29222)
xiongjiwei Dec 1, 2021
4e61d16
topsql: reduce data race of sql digest (#30296)
mornyx Dec 1, 2021
f0e81a9
ddl: Do not consider the clustered index when checking the length of …
TonsnakeLin Dec 1, 2021
7124e62
*: update client-go to use resolveForRead (#30314)
youjiali1995 Dec 1, 2021
b5cb933
ddl/ddl_algorithm_test.go: migrate test-infra to testify (#30268)
HenryZhang-ZHY Dec 1, 2021
0a92c4d
store: Add metrics for pd api call time (#30062)
jyz0309 Dec 1, 2021
94be83b
*: replace compareDatum by compare and fix wrong optimize order by (#…
wjhuang2016 Dec 1, 2021
fa686a9
metrics/grafana: fix display for 'Start TSO Wait Duration' panel (#30…
tiancaiamao Dec 1, 2021
92c7c07
ddl: migrate test-infra to testify for ddl/options_test.go (#30269)
tongtongyin Dec 1, 2021
4785dc4
planner: fix inconsistent schema between UnionAll and child operator …
time-and-fate Dec 1, 2021
7746b6b
test: fix incorrect regexp pattern during migrating test (#30080)
sleepymole Dec 1, 2021
124f1fc
br: add more precise check for lock file (#30218)
3pointer Dec 2, 2021
450d72f
executor: replace should not change other rows when auto ID is out of…
tangenta Dec 2, 2021
04a387c
tidb-server: return 1 while failing to set cpu affinity (#30197)
zhangyangyu Dec 2, 2021
7acb80f
executor: migrate test-infra to testify for executor/index_lookup_joi…
hawkingrei Dec 2, 2021
4947cf1
expression: fix wrong result of greatest/least(mixed unsigned/signed …
sylzd Dec 2, 2021
8e2b962
types: casting JSON literal `null` to number types should fail (#30278)
djshow832 Dec 2, 2021
732b7eb
executor: fix data race in oomtest (#30339)
hawkingrei Dec 2, 2021
6898975
parser: fix missing charset and collation of handle column (#30320)
wjhuang2016 Dec 2, 2021
004382d
expression: don't append null when param for char() is null (#30236)
Defined2014 Dec 2, 2021
6475e89
*: add warn log for stale read (#30340)
Yisaer Dec 2, 2021
2242f9c
parser: support multi bracket for subquery (#29656)
sylzd Dec 2, 2021
cbe5240
session, sessionctx/variable: fix validation recursion bug (#30293)
morgo Dec 2, 2021
dae711c
planner: Add trace for agg pushdown rule (#30262)
Yisaer Dec 2, 2021
cec4acb
planner/core: fix a data race when building plan for cached table (#3…
tiancaiamao Dec 2, 2021
9aa7563
executor: avoid sum from avg overflow (#30010)
fzhedu Dec 3, 2021
891517f
config: make EnableSlowLog atomic (#30346)
hawkingrei Dec 3, 2021
ac69c60
*: fix goroutine leak in ddl intergration test (#30369)
lcwangchao Dec 3, 2021
4fbbd5a
executor: make projection executor unparallel for insert/update/delet…
tiancaiamao Dec 3, 2021
70d2973
executor, util: reset offsets and nullBitMap for MutRow when setting …
XuHuaiyu Dec 3, 2021
2844660
expression: fix tidb can't alter table from other-type with null valu…
AilinKid Dec 3, 2021
e3e2e8d
dumpling: fix default collation with upstream when dump database and …
WizardXiao Dec 4, 2021
ffe9b38
ddl: fix the enum default value by triming trailing space (#30356)
sylzd Dec 6, 2021
9c62e96
expression: migrate test-infra to testify for flag_simplify_test.go (…
tisonkun Dec 6, 2021
d3ba0c2
server: refine code logic in handleDownloadFile (#30422)
Yisaer Dec 6, 2021
0b3ec09
ddl: migrate test-infra to testify for ddl/table_test.go (#30267)
hawkingrei Dec 6, 2021
464db6b
ddl: handle the error from `addBatchDDLJobs()` correctly (#30401)
tangenta Dec 6, 2021
7203707
br: fix the integration tests (#30423)
3pointer Dec 6, 2021
dfd7157
util, cmd: remove unused filesort (#30438)
wjhuang2016 Dec 6, 2021
3131085
*: update client-go for small backoff time (#30436)
youjiali1995 Dec 6, 2021
d7c2333
server: Fix unstable tests with FakeAuthSwitch (#30287)
dveeden Dec 6, 2021
9d307b0
dumpling: fix dump failed when sequence exists (#30164)
sylzd Dec 6, 2021
d3c6045
*: replace compareDatum by compare (#30421)
wjhuang2016 Dec 6, 2021
bc08b50
lightning: fix gcs max key limit (#30393)
Leavrth Dec 6, 2021
3337405
expression, parser: add built-in func is_uuid (#30318)
unconsolable Dec 6, 2021
889f001
expression: migrate test-infra to testify for constant_fold_test.go (…
tisonkun Dec 7, 2021
cb88a4c
executor: fix pipelined window invalid memory address (#30418)
wshwsh12 Dec 7, 2021
b2ea0e6
makefile: add gotestsum for verify ci (#29848)
tangenta Dec 7, 2021
53f33fa
server: close sql rows to fix unstable test (#30306)
hawkingrei Dec 7, 2021
239fca2
Makefile: add coverage record for BR and Dumpling (#30457)
tangenta Dec 7, 2021
181c0c9
executor: track the mem usage of IndexMergeReader (#30210)
guo-shaoge Dec 7, 2021
7d4895f
infosync: close body when ReadAll encounters error (#30462)
rleungx Dec 7, 2021
ffd59ec
planner: show accessed partition when explain mpp query over partitio…
windtalker Dec 7, 2021
376d542
*: Fix use of user identity in SHOW GRANTS + error messages (#30294)
morgo Dec 7, 2021
4fb333c
ddl: add not null flag for auto_increment column (#30477)
wjhuang2016 Dec 7, 2021
6eb14e0
expression: make some unstable test serial (#30323)
hawkingrei Dec 7, 2021
e7aeae7
expression: migrate test-infra to testify for constant_propagation_te…
tisonkun Dec 7, 2021
2af4624
executor: stable test TestSetDDLReorgBatchSize and TestSetDDLReorgWor…
wjhuang2016 Dec 7, 2021
f82ca6a
statistics, util/ranger: add cardinality estimation trace for `GetRow…
time-and-fate Dec 7, 2021
b585bf9
*: skip mysql client goroutine leak detection in integration ddl (#30…
lcwangchao Dec 7, 2021
d83ee8c
executor,util: write slow query to slow log no matter what log level …
TonsnakeLin Dec 7, 2021
27f7b59
executor: enable index_merge used in transaction. (#29875)
guo-shaoge Dec 8, 2021
7a5a8bf
logutil: add testcase for SlowQueryLogger.MaxDays/MaxSize/MaxBackups …
guo-shaoge Dec 8, 2021
daf8f3e
expression: fix data race in builtin_other_vec_generated_test.go (#30…
hawkingrei Dec 8, 2021
2a1ea89
expression: fix data race in the collationInfo (#30490)
hawkingrei Dec 8, 2021
e520e46
planner/core, session: fix error message of wrong variable scope (#30…
morgo Dec 8, 2021
dac5a68
lightning: support Re/ReregisterMySQL by different tls name (#30463)
Ehco1996 Dec 8, 2021
66dfc70
executor: TestBatchGetandPointGetwithHashPartition test typo (#29669)…
mjonss Dec 8, 2021
1d51261
mockstore: improve log to avoid panic for nil pointer (#30513)
hawkingrei Dec 8, 2021
9773f83
*: replace compareDatum by compare, PR 10 (#30456)
wjhuang2016 Dec 8, 2021
6229c89
planner: Disable dynamic partition prune mode for all non-autocommit …
mjonss Dec 8, 2021
e15e457
expression: change the log level of an confusing log from warn to deb…
XuHuaiyu Dec 8, 2021
7aee819
br: Check crypter.key valid before backup (#29991)
joccau Dec 9, 2021
d9ef107
*: replace compareDatum by compare, PR 11 (#30465)
wjhuang2016 Dec 9, 2021
9f744cd
dumpling: fix default column collation with upstream when dump table …
WizardXiao Dec 9, 2021
6a43317
server: fix prepared cursor select (#30285)
xhebox Dec 9, 2021
80ea238
executor: HashJoinExec checks the buildError even if the probeSide is…
XuHuaiyu Dec 9, 2021
05f7c6d
parser, expression: follow mysql, increase interval precedence (#30528)
Defined2014 Dec 9, 2021
1a543c4
makefile: set timeout 25m for make race (#30555)
hawkingrei Dec 9, 2021
b6c45af
planner: fix the unstable test TestAnalyzeGlobalStatsWithOpts/2 (#30576)
qw4990 Dec 9, 2021
8bc22b1
expression,types: Adjusts UNIX_TIMESTAMP() for non-existing DST value…
mjonss Dec 9, 2021
1833b61
br: add res.Body.close to avoid leak (#30545)
hawkingrei Dec 9, 2021
c89c473
lightning: add back integration test lightning_error_summary (#30547)
glorv Dec 9, 2021
8db9454
sessionctx/variable: small refactor (split large file) (#30511)
morgo Dec 9, 2021
fbcf757
ddl: let `admin cancel ddl jobs` run in a new transaction (#30549)
djshow832 Dec 9, 2021
89fd697
*: Retry when placement PutBundles failed (#30590)
lcwangchao Dec 10, 2021
7c416ed
dumpling: delete unit test in github actions (#30562)
lichunzhu Dec 10, 2021
23af09d
*: support trace plan target='estimation' statement (#30491)
time-and-fate Dec 10, 2021
ed384b2
expression: migrate test-infra to testify for integration_test.go (#3…
hawkingrei Dec 10, 2021
828c1dd
planner: support trace for min/max eliminate (#30441)
Yisaer Dec 10, 2021
4f42291
br: remove cdclog in br (#30573)
joccau Dec 10, 2021
aa002f2
*: show cmd to check if all needed histograms are loaded (#29672)
chrysan Dec 10, 2021
03b6a8e
expression: clone repertoire when clone the scalar function (#30602)
xiongjiwei Dec 10, 2021
9a074f1
*: use the real StateRemote interface implementation for cached table…
tiancaiamao Dec 10, 2021
9d4da4f
*: query failed after add index / timestamp out-of-range (#28424) (#2…
mjonss Dec 10, 2021
512373e
planner: implement collecting predicate columns from logical plan (#2…
xuyifangreeneyes Dec 10, 2021
c08f7fc
*: show PK name when decoding the clustered index row key (#30623)
JmPotato Dec 10, 2021
66163de
ddl/callback_test.go: migrate test-infra to testify (#30317)
zach030 Dec 10, 2021
93e2c12
*: Rename some names of placement ddl operation (#30622)
lcwangchao Dec 10, 2021
3dce612
executor: fix data race in the index_lookup_hash_join (#30619)
hawkingrei Dec 11, 2021
8e525d8
ddl: remove unnecessary locking when adding an index (#29772)
wjhuang2016 Dec 11, 2021
626dee7
server: try to make `TidbTestSuite` more stable (#30643)
bb7133 Dec 12, 2021
7555536
*: Add some PD tests for placement and fix some bug found (#30621)
lcwangchao Dec 12, 2021
1f26870
*: migrate sync.WaitGroup to util.WaitGroupWrapper (#30644)
hawkingrei Dec 13, 2021
6495a5a
planner: add trace for join eliminate rule (#30343)
Yisaer Dec 13, 2021
9a64217
executor: migrate test-infra to testify for executor/shuffle_test.go…
tongtongyin Dec 13, 2021
b8dcc09
planner: make (*AccessPath).OnlyPointRange more succinct (#30520)
xuyifangreeneyes Dec 13, 2021
a532973
planner: add trace for join reorder (#30394)
Yisaer Dec 13, 2021
d3833c2
executor: migrate test-infra to testify for executor/union_scan_test.…
tongtongyin Dec 13, 2021
e8577c7
expression: make cast return error if cast binary literal to another …
xiongjiwei Dec 13, 2021
68118f1
*: update tikv client (#30670)
Yisaer Dec 13, 2021
d8a48d9
*: update sysutil in go.mod to fix panic when search log (#30523)
crazycs520 Dec 13, 2021
12e2288
topsql: shouldn't evict the SQL meta, since the evicted SQL can be ap…
crazycs520 Dec 13, 2021
b876788
testify: migrate test-infra to testify for analyze_test.go (#30640)
znhh6018 Dec 13, 2021
b626745
util: replace compareDatum by compare, point part (#30575)
wjhuang2016 Dec 13, 2021
b4df7ca
test: make all the tests run in serial (#30692)
tangenta Dec 14, 2021
ac25e9f
statistics: add mutex for Handle.globalMap and Handle.feedback (#30550)
xuyifangreeneyes Dec 14, 2021
a8ad203
executor: fix regular expression in json so that it could match ident…
znhh6018 Dec 14, 2021
dd7f71f
util/testkit/testkit.go: fix typo (#30638)
eddiearc Dec 14, 2021
c30d34f
planner: Introduce a new global variable to control the historical st…
An-DJ Dec 14, 2021
e9b1fb8
topsql: introduce datasink interface (#30662)
zhongzc Dec 14, 2021
2f42f7c
planner: unify the argument of stats functions to use SessionCtx inst…
qw4990 Dec 14, 2021
5d463f3
metrics: fix the Max SafeTS Gap metrics (#30689)
JmPotato Dec 14, 2021
4b48e55
lightning: Add source dir existence check for s3 (#30674)
glorv Dec 14, 2021
950a274
golangci-lint: support durationcheck (#30027)
wangggong Dec 14, 2021
6c0fcea
executor: fix data race on IndexHashJoin.cancelFunc (#30701)
XuHuaiyu Dec 14, 2021
813f6ef
sessionctx/variable: change tidb_store_limit to global only (#30522)
morgo Dec 15, 2021
0d0805d
statistics: remove reassignment of Handle.pool in NewHandle (#30675)
xuyifangreeneyes Dec 15, 2021
93b0a58
br: fix some unstable unit test cases. (#30716)
3pointer Dec 15, 2021
22418cd
bindinfo: fix the comment typo (#30616)
zhangjinpeng87 Dec 15, 2021
04a9618
server: support decoding prepared string args to character_set_client…
tangenta Dec 15, 2021
3fe9263
expression: fix enum type join binary get wrong result (#30445)
sylzd Dec 15, 2021
64f7309
cmd/explaintest: fix wrong result comparison for explain test (#30717)
xuyifangreeneyes Dec 15, 2021
91ceec5
parallel create tables in br
fengou1 Dec 15, 2021
87dbcc2
metrics: fix copr-cache metrics (#30712)
jackysp Dec 15, 2021
2e79433
test: merge executor's serial tests to other tests (#30711)
tangenta Dec 15, 2021
14aca77
ddl: add batch create table api
xhebox Oct 12, 2021
73a7b8a
ddl: add unit tests
xhebox Oct 12, 2021
dee6c35
ddl: fix fmt
xhebox Oct 12, 2021
5b46f1b
ddl: typo
xhebox Oct 18, 2021
88a0e3f
ddl: fix tests
xhebox Oct 19, 2021
5c50048
ddl: rename to BatchCreateTableWithInfo
xhebox Oct 20, 2021
d35fc31
ddl: trace the error
xhebox Oct 20, 2021
f43fa0f
ddl: comments
xhebox Oct 20, 2021
7348348
ddl: cancle the job right
xhebox Oct 20, 2021
ad5812e
ddl: cancel the job right 2
xhebox Oct 25, 2021
1b70b8b
ddl: report error if entry too large
xhebox Oct 26, 2021
05aa57c
ddl: report error when table is duplicated
xhebox Oct 27, 2021
e4582a7
ddl: go fmt
xhebox Oct 27, 2021
7913c49
infoschema: improve batch memory perf
xhebox Oct 27, 2021
84bdeaf
ddl: retain ID
xhebox Oct 29, 2021
bd0b779
ddl: reduce log frequency
xhebox Dec 15, 2021
c5e7cb8
ddl: fix tests
xhebox Dec 15, 2021
497bc0f
fix conflict
fengou1 Dec 21, 2021
ced6a21
fix merge issue
fengou1 Dec 21, 2021
501fc52
fix issue that loss table restore
fengou1 Dec 22, 2021
17bd4dd
refactoring code
fengou1 Dec 22, 2021
4115494
refactoring code
fengou1 Dec 22, 2021
6d971dc
Merge branch 'br_ddl' into merge_ddl
fengou1 Dec 26, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion br/pkg/glue/glue.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import (

// interface is to bulk create table parallelly
type BulkCreateTableSession interface {
CreateTables(ctx context.Context, tables map[string][]*model.TableInfo, batchDdlSize uint) error
CreateTables(ctx context.Context, tables map[string][]*model.TableInfo) error
}

// Glue is an abstraction of TiDB function calls used in BR.
Expand Down
7 changes: 4 additions & 3 deletions br/pkg/gluetidb/glue.go
Original file line number Diff line number Diff line change
Expand Up @@ -131,9 +131,9 @@ func (gs *tidbSession) CreateDatabase(ctx context.Context, schema *model.DBInfo)
}

// CreateTable implements glue.Session.
func (gs *tidbSession) CreateTables(ctx context.Context, tables map[string][]*model.TableInfo, batchDdlSize uint) error {
func (gs *tidbSession) CreateTables(ctx context.Context, tables map[string][]*model.TableInfo) error {
d := domain.GetDomain(gs.se).DDL()
log.Info("tidb start create tables", zap.Uint("batchDdlSize", batchDdlSize))
log.Info("tidb start create tables")
var dbName model.CIStr
cloneTables := make([]*model.TableInfo, 0, len(tables))

Expand All @@ -159,7 +159,8 @@ func (gs *tidbSession) CreateTables(ctx context.Context, tables map[string][]*mo
cloneTables = append(cloneTables, table)
}
gs.se.SetValue(sessionctx.QueryString, queryBuilder.String())
err := d.BatchCreateTableWithInfo(gs.se, dbName, cloneTables, ddl.OnExistIgnore, true)
err := d.BatchCreateTableWithInfo(gs.se, dbName, cloneTables, ddl.OnExistIgnore)

if err != nil {
log.Info("Bulk create table from tidb failure, it possible caused by version mismatch with BR.", zap.String("Error", err.Error()))
return err
Expand Down
226 changes: 158 additions & 68 deletions br/pkg/restore/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (
"sort"
"strconv"
"strings"
"sync"
"time"

"github.com/opentracing/opentracing-go"
Expand Down Expand Up @@ -81,6 +82,7 @@ type Client struct {

restoreStores []uint64

cipher *backuppb.CipherInfo
storage storage.ExternalStorage
backend *backuppb.StorageBackend
switchModeInterval time.Duration
Expand Down Expand Up @@ -135,6 +137,10 @@ func (rc *Client) SetRateLimit(rateLimit uint64) {
rc.rateLimit = rateLimit
}

func (rc *Client) SetCrypter(crypter *backuppb.CipherInfo) {
rc.cipher = crypter
}

// SetStorage set ExternalStorage for client.
func (rc *Client) SetStorage(ctx context.Context, backend *backuppb.StorageBackend, opts *storage.ExternalStorageOptions) error {
var err error
Expand Down Expand Up @@ -415,7 +421,7 @@ func (rc *Client) createTables(
if rc.IsSkipCreateSQL() {
log.Info("skip create table and alter autoIncID")
} else {
err := db.CreateTables(ctx, tables, rc.GetBatchDdlSize())
err := db.CreateTables(ctx, tables)
if err != nil {
return nil, errors.Trace(err)
}
Expand Down Expand Up @@ -451,11 +457,12 @@ func (rc *Client) createTable(
dom *domain.Domain,
table *metautil.Table,
newTS uint64,
ddlTables map[UniqueTableName]bool,
) (CreatedTable, error) {
if rc.IsSkipCreateSQL() {
log.Info("skip create table and alter autoIncID", zap.Stringer("table", table.Info.Name))
} else {
err := db.CreateTable(ctx, table)
err := db.CreateTable(ctx, table, ddlTables)
if err != nil {
return CreatedTable{}, errors.Trace(err)
}
Expand Down Expand Up @@ -494,62 +501,71 @@ func (rc *Client) GoCreateTables(
// Could we have a smaller size of tables?
log.Info("start create tables")

ddlTables := rc.DDLJobsMap()
if span := opentracing.SpanFromContext(ctx); span != nil && span.Tracer() != nil {
span1 := span.Tracer().StartSpan("Client.GoCreateTables", opentracing.ChildOf(span.Context()))
defer span1.Finish()
ctx = opentracing.ContextWithSpan(ctx, span1)
}
outCh := make(chan CreatedTable, len(tables))
rater := logutil.TraceRateOver(logutil.MetricTableCreatedCounter)
err := rc.createTablesInWorkerPool(ctx, dom, tables, dbPool, newTS, outCh)
//cts, err := rc.createTables(ctx, rc.db, dom, tables, newTS)

if err == nil {
defer close(outCh)
// fall back to old create table (sequential create table)
} else if strings.Contains(err.Error(), "[ddl:8204]invalid ddl job") {
log.Info("fall back to the old DDL way to create table.")
createOneTable := func(c context.Context, db *DB, t *metautil.Table) error {
select {
case <-c.Done():
return c.Err()
default:
}
rt, err := rc.createTable(c, db, dom, t, newTS)
if err != nil {
log.Error("create table failed",
zap.Error(err),
zap.Stringer("db", t.DB.Name),
zap.Stringer("table", t.Info.Name))
return errors.Trace(err)
}
log.Debug("table created and send to next",
zap.Int("output chan size", len(outCh)),
zap.Stringer("table", t.Info.Name),
zap.Stringer("database", t.DB.Name))
outCh <- rt
rater.Inc()
rater.L().Info("table created",
zap.Stringer("table", t.Info.Name),
zap.Stringer("database", t.DB.Name))
return nil
}
go func() {
var err error = nil
if rc.batchDllSize > 0 {
err = rc.createTablesInWorkerPool(ctx, dom, tables, dbPool, newTS, outCh)

if err == nil {
log.Info("bulk to create tables success.")
defer close(outCh)
defer log.Debug("all tables are created")
var err error
if len(dbPool) > 0 {
err = rc.createTablesWithDBPool(ctx, createOneTable, tables, dbPool)
} else {
err = rc.createTablesWithSoleDB(ctx, createOneTable, tables)
}
if err != nil {
errCh <- err
}
}()
} else {
errCh <- err
// fall back to old create table (sequential create table)
} else if strings.Contains(err.Error(), "[ddl:8204]invalid ddl job") {
log.Info("fall back to the old DDL way to create table.")
} else {
log.Error("bulk to create tables failure.")
errCh <- err
return outCh
}
}

createOneTable := func(c context.Context, db *DB, t *metautil.Table) error {
select {
case <-c.Done():
return c.Err()
default:

}
rt, err := rc.createTable(c, db, dom, t, newTS, ddlTables)
if err != nil {
log.Error("create table failed",
zap.Error(err),
zap.Stringer("db", t.DB.Name),
zap.Stringer("table", t.Info.Name))
return errors.Trace(err)
}
log.Debug("table created and send to next",
zap.Int("output chan size", len(outCh)),
zap.Stringer("table", t.Info.Name),
zap.Stringer("database", t.DB.Name))
outCh <- rt
rater.Inc()
rater.L().Info("table created",
zap.Stringer("table", t.Info.Name),
zap.Stringer("database", t.DB.Name))
return nil
}
go func() {
defer close(outCh)
defer log.Debug("all tables are created")
var err error
if len(dbPool) > 0 {
err = rc.createTablesWithDBPool(ctx, createOneTable, tables, dbPool)
} else {
err = rc.createTablesWithSoleDB(ctx, createOneTable, tables)
}
if err != nil {
errCh <- err
}
}()
return outCh
}

Expand Down Expand Up @@ -585,7 +601,9 @@ func (rc *Client) createTablesInWorkerPool(ctx context.Context, dom *domain.Doma
workers := utils.NewWorkerPool(uint(len(dbPool)), "Create Tables Worker")
numOfTables := len(tables)
lastSent := 0
for i := int(rc.batchDllSize); i <= numOfTables; i = i + int(rc.batchDllSize) {

for i := int(rc.batchDllSize); i < numOfTables+int(rc.batchDllSize); i = i + int(rc.batchDllSize) {

log.Info("create tables", zap.Int("table start", lastSent), zap.Int("table end", i))
if i > numOfTables {
i = numOfTables
Expand Down Expand Up @@ -727,7 +745,7 @@ func (rc *Client) RestoreFiles(
zap.Duration("take", time.Since(fileStart)))
updateCh.Inc()
}()
return rc.fileImporter.Import(ectx, filesReplica, rewriteRules)
return rc.fileImporter.Import(ectx, filesReplica, rewriteRules, rc.cipher)
})
}

Expand Down Expand Up @@ -768,7 +786,7 @@ func (rc *Client) RestoreRaw(
rc.workerPool.ApplyOnErrorGroup(eg,
func() error {
defer updateCh.Inc()
return rc.fileImporter.Import(ectx, []*backuppb.File{fileReplica}, EmptyRewriteRule())
return rc.fileImporter.Import(ectx, []*backuppb.File{fileReplica}, EmptyRewriteRule(), rc.cipher)
})
}
if err := eg.Wait(); err != nil {
Expand Down Expand Up @@ -844,6 +862,8 @@ func (rc *Client) switchTiKVMode(ctx context.Context, mode import_sstpb.SwitchMo
gctx,
store.GetAddress(),
opt,
grpc.WithBlock(),
grpc.FailOnNonTempDialError(true),
grpc.WithConnectParams(grpc.ConnectParams{Backoff: bfConf}),
// we don't need to set keepalive timeout here, because the connection lives
// at most 5s. (shorter than minimal value for keepalive time!)
Expand Down Expand Up @@ -880,17 +900,25 @@ func (rc *Client) GoValidateChecksum(
) <-chan struct{} {
log.Info("Start to validate checksum")
outCh := make(chan struct{}, 1)
wg := new(sync.WaitGroup)
wg.Add(2)
loadStatCh := make(chan *CreatedTable, 1024)
// run the stat loader
go func() {
defer wg.Done()
rc.updateMetaAndLoadStats(ctx, loadStatCh)
}()
workers := utils.NewWorkerPool(defaultChecksumConcurrency, "RestoreChecksum")
go func() {
wg, ectx := errgroup.WithContext(ctx)
eg, ectx := errgroup.WithContext(ctx)
defer func() {
log.Info("all checksum ended")
if err := wg.Wait(); err != nil {
if err := eg.Wait(); err != nil {
errCh <- err
}
outCh <- struct{}{}
close(outCh)
close(loadStatCh)
wg.Done()
}()

for {
select {
// if we use ectx here, maybe canceled will mask real error.
Expand All @@ -900,14 +928,14 @@ func (rc *Client) GoValidateChecksum(
if !ok {
return
}
workers.ApplyOnErrorGroup(wg, func() error {

workers.ApplyOnErrorGroup(eg, func() error {
start := time.Now()
defer func() {
elapsed := time.Since(start)
summary.CollectDuration("restore checksum", elapsed)
summary.CollectSuccessUnit("table checksum", 1, elapsed)
}()
err := rc.execChecksum(ectx, tbl, kvClient, concurrency)
err := rc.execChecksum(ectx, tbl, kvClient, concurrency, loadStatCh)
if err != nil {
return errors.Trace(err)
}
Expand All @@ -917,10 +945,21 @@ func (rc *Client) GoValidateChecksum(
}
}
}()
go func() {
wg.Wait()
log.Info("all checksum ended")
close(outCh)
}()
return outCh
}

func (rc *Client) execChecksum(ctx context.Context, tbl CreatedTable, kvClient kv.Client, concurrency uint) error {
func (rc *Client) execChecksum(
ctx context.Context,
tbl CreatedTable,
kvClient kv.Client,
concurrency uint,
loadStatCh chan<- *CreatedTable,
) error {
logger := log.With(
zap.String("db", tbl.OldTable.DB.Name.O),
zap.String("table", tbl.OldTable.Info.Name.O),
Expand Down Expand Up @@ -969,16 +1008,49 @@ func (rc *Client) execChecksum(ctx context.Context, tbl CreatedTable, kvClient k
)
return errors.Annotate(berrors.ErrRestoreChecksumMismatch, "failed to validate checksum")
}
if table.Stats != nil {
logger.Info("start loads analyze after validate checksum",
zap.Int64("old id", tbl.OldTable.Info.ID),
zap.Int64("new id", tbl.Table.ID),
)
if err := rc.statsHandler.LoadStatsFromJSON(rc.dom.InfoSchema(), table.Stats); err != nil {
logger.Error("analyze table failed", zap.Any("table", table.Stats), zap.Error(err))

loadStatCh <- &tbl
return nil
}

func (rc *Client) updateMetaAndLoadStats(ctx context.Context, input <-chan *CreatedTable) {
for {
select {
case <-ctx.Done():
return
case tbl, ok := <-input:
if !ok {
return
}

// Not need to return err when failed because of update analysis-meta
restoreTS, err := rc.GetTS(ctx)
if err != nil {
log.Error("getTS failed", zap.Error(err))
} else {
err = rc.db.UpdateStatsMeta(ctx, tbl.Table.ID, restoreTS, tbl.OldTable.TotalKvs)
if err != nil {
log.Error("update stats meta failed", zap.Any("table", tbl.Table), zap.Error(err))
}
}

table := tbl.OldTable
if table.Stats != nil {
log.Info("start loads analyze after validate checksum",
zap.Int64("old id", tbl.OldTable.Info.ID),
zap.Int64("new id", tbl.Table.ID),
)
start := time.Now()
if err := rc.statsHandler.LoadStatsFromJSON(rc.dom.InfoSchema(), table.Stats); err != nil {
log.Error("analyze table failed", zap.Any("table", table.Stats), zap.Error(err))
}
log.Info("restore stat done",
zap.String("table", table.Info.Name.L),
zap.String("db", table.DB.Name.L),
zap.Duration("cost", time.Since(start)))
}
}
}
return nil
}

const (
Expand Down Expand Up @@ -1155,6 +1227,24 @@ func (rc *Client) IsSkipCreateSQL() bool {
return rc.noSchema
}

// DDLJobsMap returns a map[UniqueTableName]bool about < db table, hasCreate/hasTruncate DDL >.
// if we execute some DDLs before create table.
// we may get two situation that need to rebase auto increment/random id.
// 1. truncate table: truncate will generate new id cache.
// 2. create table/create and rename table: the first create table will lock down the id cache.
// because we cannot create onExistReplace table.
// so the final create DDL with the correct auto increment/random id won't be executed.
func (rc *Client) DDLJobsMap() map[UniqueTableName]bool {
m := make(map[UniqueTableName]bool)
for _, job := range rc.ddlJobs {
switch job.Type {
case model.ActionTruncateTable, model.ActionCreateTable, model.ActionRenameTable:
m[UniqueTableName{job.SchemaName, job.BinlogInfo.TableInfo.Name.String()}] = true
}
}
return m
}

// PreCheckTableTiFlashReplica checks whether TiFlash replica is less than TiFlash node.
func (rc *Client) PreCheckTableTiFlashReplica(
ctx context.Context,
Expand Down
Loading