Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve](group_commit) optimize group commit select be logic #28190

Merged
merged 1 commit into from
Dec 8, 2023
Merged

[improve](group_commit) optimize group commit select be logic #28190

merged 1 commit into from
Dec 8, 2023

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Dec 8, 2023

Proposed changes

Group commit choose be always first no decommissioned be in all be.

Choose be with selectBackendIdsByPolicy like common stream load and do not choose decommissioned be may be better.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@sollhui
Copy link
Contributor Author

sollhui commented Dec 8, 2023

run buildall

Copy link
Contributor

@mymeiyi mymeiyi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Dec 8, 2023

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit b0a56e296f0844dd68160a732dabf7f4283765dc, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4688	4432	4496	4432
q2	357	162	205	162
q3	1437	1173	1216	1173
q4	1106	903	968	903
q5	3126	3118	3141	3118
q6	249	130	129	129
q7	974	479	479	479
q8	2169	2244	2150	2150
q9	6663	6630	6641	6630
q10	3201	3259	3249	3249
q11	325	200	202	200
q12	351	207	208	207
q13	4548	3777	3791	3777
q14	244	211	215	211
q15	571	516	527	516
q16	440	381	386	381
q17	1007	588	599	588
q18	7528	7147	7334	7147
q19	1503	1375	1387	1375
q20	544	319	314	314
q21	3011	2594	2586	2586
q22	347	277	277	277
Total cold run time: 44389 ms
Total hot run time: 40004 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4371	4365	4391	4365
q2	270	167	178	167
q3	3514	3510	3501	3501
q4	2368	2355	2346	2346
q5	5732	5696	5730	5696
q6	240	123	123	123
q7	2362	1879	1864	1864
q8	3493	3491	3480	3480
q9	9034	9028	9020	9020
q10	3897	3973	3964	3964
q11	512	379	380	379
q12	769	606	588	588
q13	4280	3568	3561	3561
q14	285	264	262	262
q15	564	523	518	518
q16	487	461	478	461
q17	1853	1836	1874	1836
q18	8654	8210	8348	8210
q19	1706	1718	1744	1718
q20	2246	1955	1932	1932
q21	6451	6139	6131	6131
q22	501	421	407	407
Total cold run time: 63589 ms
Total hot run time: 60529 ms

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Dec 8, 2023

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 8, 2023
@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.64 seconds
stream load tsv: 598 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17211934939 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit b0a56e296f0844dd68160a732dabf7f4283765dc, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4689	4416	4444	4416
q2	361	153	159	153
q3	1451	1245	1167	1167
q4	1100	895	825	825
q5	3104	3115	3222	3115
q6	256	131	128	128
q7	1034	494	477	477
q8	2195	2213	2156	2156
q9	6715	6655	6615	6615
q10	3196	3233	3251	3233
q11	330	193	199	193
q12	360	208	205	205
q13	4517	3825	3830	3825
q14	246	212	211	211
q15	563	517	536	517
q16	448	386	389	386
q17	997	548	545	545
q18	7478	7432	6991	6991
q19	1529	1358	1447	1358
q20	521	325	304	304
q21	3042	2583	2633	2583
q22	345	274	283	274
Total cold run time: 44477 ms
Total hot run time: 39677 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4395	4385	4368	4368
q2	263	166	173	166
q3	3523	3504	3506	3504
q4	2362	2362	2353	2353
q5	5710	5704	5700	5700
q6	238	122	125	122
q7	2387	1863	1888	1863
q8	3484	3490	3497	3490
q9	9017	8980	9002	8980
q10	3878	3976	3981	3976
q11	495	380	377	377
q12	772	580	586	580
q13	4284	3597	3520	3520
q14	278	248	244	244
q15	576	520	512	512
q16	502	453	463	453
q17	1867	1844	1863	1844
q18	8657	8221	8172	8172
q19	1703	1716	1725	1716
q20	2235	1932	1946	1932
q21	6465	6125	6084	6084
q22	491	403	416	403
Total cold run time: 63582 ms
Total hot run time: 60359 ms

@yiguolei yiguolei merged commit 9d9b646 into apache:master Dec 8, 2023
@sollhui sollhui deleted the group_commit_select_be branch December 11, 2023 03:08
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
…28190

Group commit choose be always first no decommissioned be in all be.

Choose be with selectBackendIdsByPolicy like common stream load and do not choose decommissioned be may be better.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants