Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](partial update) Fix some bugs about partial update #28358

Merged
merged 3 commits into from
Dec 14, 2023
Merged

[fix](partial update) Fix some bugs about partial update #28358

merged 3 commits into from
Dec 14, 2023

Conversation

bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Dec 13, 2023 •
edited
Loading

Proposed changes

fix some bugs about partial update, related pr:

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 13, 2023

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.67 seconds
stream load tsv: 587 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17219742996 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit c9bafb5c1aea94be290746c95ba38ab9d14a12ce, data reload: false

run tpch-sf100 query with default conf and session variables
q1	50015	31863	31837	31837
q2	366	137	147	137
q3	1515	1223	1252	1223
q4	1099	909	880	880
q5	3189	3161	3164	3161
q6	262	147	149	147
q7	1007	501	485	485
q8	2162	2208	2204	2204
q9	6881	6846	6859	6846
q10	3247	3311	3296	3296
q11	334	217	203	203
q12	353	206	202	202
q13	4548	3817	3756	3756
q14	431	317	319	317
q15	792	1318	1352	1318
q16	443	385	384	384
q17	1015	585	596	585
q18	7259	6938	7234	6938
q19	1531	1444	1441	1441
q20	613	294	298	294
q21	3050	2640	2677	2640
q22	343	282	291	282
Total cold run time: 90455 ms
Total hot run time: 68576 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	30822	31865	31819	31819
q2	278	163	171	163
q3	3583	3546	3568	3546
q4	2396	2367	2377	2367
q5	5754	5744	5757	5744
q6	240	133	131	131
q7	3055	3419	3363	3363
q8	3529	3533	3545	3533
q9	9164	9158	9088	9088
q10	3962	4046	4021	4021
q11	825	904	1368	904
q12	764	601	599	599
q13	4297	3588	3593	3588
q14	512	465	456	456
q15	1098	1320	1329	1320
q16	498	440	463	440
q17	1890	1851	1869	1851
q18	8727	7967	8224	7967
q19	1756	1762	1780	1762
q20	2890	2614	2663	2614
q21	6482	6111	6111	6111
q22	509	426	423	423
Total cold run time: 93031 ms
Total hot run time: 91810 ms

@bobhan1 bobhan1 force-pushed the fix-partial-update-bugs branch from c9bafb5 to 3d9901e Compare December 14, 2023 06:57
@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 14, 2023

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@bobhan1 bobhan1 force-pushed the fix-partial-update-bugs branch from 3d9901e to 49d74fb Compare December 14, 2023 07:13
@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 14, 2023

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 49d74fb39b2e694ca7fc65c26772e024cac64ce2, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4702	4449	4446	4446
q2	371	146	158	146
q3	1458	1197	1176	1176
q4	1116	910	881	881
q5	3137	3139	3141	3139
q6	252	129	128	128
q7	987	492	484	484
q8	2172	2201	2177	2177
q9	6649	6705	6610	6610
q10	3206	3274	3264	3264
q11	324	216	204	204
q12	361	201	210	201
q13	4570	3837	3835	3835
q14	243	213	209	209
q15	566	520	540	520
q16	433	384	388	384
q17	996	580	564	564
q18	7253	7033	6901	6901
q19	1529	1365	1388	1365
q20	524	300	302	300
q21	3086	2647	2649	2647
q22	349	284	295	284
Total cold run time: 44284 ms
Total hot run time: 39865 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4398	4394	4372	4372
q2	269	165	176	165
q3	3539	3524	3515	3515
q4	2409	2388	2391	2388
q5	5727	5737	5751	5737
q6	239	119	122	119
q7	2370	1869	1853	1853
q8	3503	3509	3508	3508
q9	9005	8981	9005	8981
q10	3925	3985	4011	3985
q11	503	390	381	381
q12	752	599	584	584
q13	4297	3555	3529	3529
q14	284	264	253	253
q15	579	517	522	517
q16	526	483	490	483
q17	1853	1858	1859	1858
q18	8672	8121	8273	8121
q19	1735	1715	1714	1714
q20	2251	1962	1933	1933
q21	6532	6193	6169	6169
q22	517	413	427	413
Total cold run time: 63885 ms
Total hot run time: 60578 ms

@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 14, 2023

run buildall

@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 14, 2023

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit ae3eda3a05f11bf6f8eab9d24ecb230a10283bfd, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4691	4467	4491	4467
q2	366	149	159	149
q3	1458	1226	1239	1226
q4	1114	921	886	886
q5	3160	3181	3178	3178
q6	248	129	129	129
q7	990	485	479	479
q8	2203	2216	2176	2176
q9	6696	6667	6676	6667
q10	3211	3268	3283	3268
q11	321	202	208	202
q12	352	213	211	211
q13	4565	3812	3800	3800
q14	245	214	210	210
q15	559	525	516	516
q16	444	383	386	383
q17	1003	592	570	570
q18	7197	6940	6951	6940
q19	1508	1395	1375	1375
q20	539	299	289	289
q21	3048	2649	2660	2649
q22	354	283	289	283
Total cold run time: 44272 ms
Total hot run time: 40053 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4404	4406	4390	4390
q2	266	162	172	162
q3	3528	3536	3514	3514
q4	2392	2382	2393	2382
q5	5708	5718	5718	5718
q6	240	125	122	122
q7	2381	1857	1872	1857
q8	3519	3521	3509	3509
q9	9054	9026	8971	8971
q10	3937	3981	3984	3981
q11	507	381	382	381
q12	764	607	598	598
q13	4316	3542	3561	3542
q14	283	259	266	259
q15	576	525	515	515
q16	497	458	451	451
q17	1869	1850	1846	1846
q18	8609	8340	8133	8133
q19	1712	1741	1737	1737
q20	2233	1945	1920	1920
q21	6573	6144	6169	6144
q22	520	421	441	421
Total cold run time: 63888 ms
Total hot run time: 60553 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.37 seconds
stream load tsv: 583 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17219646050 Bytes

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 14, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

zhannngchen
zhannngchen approved these changes Dec 14, 2023
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit 415c6d8 into apache:master Dec 14, 2023
seawinde pushed a commit to seawinde/doris that referenced this pull request Dec 15, 2023
@bobhan1 @seawinde
[fix](partial update) Fix some bugs about partial update (apache#28358)
14c44de
@wm1581066 wm1581066 added the usercase Important user case type label label Dec 16, 2023
@bobhan1 bobhan1 mentioned this pull request Dec 22, 2023
Merged
xiaokang pushed a commit that referenced this pull request Dec 23, 2023
@bobhan1
[branch-2.0-pick](partial update) Pick some PRs about partial update #…
a15b289 
…28321 #28358 (#28844)
hello-stephen pushed a commit to hello-stephen/doris that referenced this pull request Dec 28, 2023
@bobhan1
[fix](partial update) Fix some bugs about partial update (apache#28358)
1323298
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@bobhan1 @HappenLee
[fix](partial update) Fix some bugs about partial update (apache#28358)
bde0a22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhannngchen zhannngchen zhannngchen approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.4-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@bobhan1 @doris-robot @zhannngchen @morrySnow @xiaokang @yiguolei @wm1581066