Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[branch-3.0](pick) Pick 6 commits #41715

Merged
merged 7 commits into from
Oct 11, 2024
Merged

Conversation

Gabriel39
Copy link
Contributor

Proposed changes

pick #39982 #41292 #40576 #41555 #41589 #41592

Unify `prepare` and `open` in `Operator`.
apache#41292)

Consider 3 pipelines in this fragment (... -> join -> shuffle) :
pipeline 0 : `... -> local exchange sink`
pipeline 1 : `... -> join build (INNER JOIN)`
pipeline 2 : `local exchange source -> join probe (INNER JOIN) -> data
stream sender `

Assume the JoinBuild returned 0 rows, join probe can finish directly
once join build finished and do not need to wait for the `local exchange
sink` finished. In this case, if pipeline 0 is blocked by a dependency
for a long time, pipeline 2 should notify pipeline 0 to finish.
…e#41555)

For plan `local exchange (hash shuffle) -> union -> colocated agg`, we
must ensure local exchange use the same hash algorithm as MPP shuffling.

This problem is covered by our test cases but only can be reproduced on
multiple BEs so no case is added in this PR.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Contributor Author

run buildall

1 similar comment
@Gabriel39
Copy link
Contributor Author

run buildall

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39976 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0841cee76005762132b4c8df3e6060a28014ffa9, data reload: false

------ Round 1 ----------------------------------
q1	17612	7436	7246	7246
q2	2038	160	154	154
q3	10709	1053	1153	1053
q4	10548	762	781	762
q5	7750	2791	2805	2791
q6	229	148	144	144
q7	999	655	607	607
q8	9572	1902	1958	1902
q9	7960	6393	6418	6393
q10	6986	2274	2269	2269
q11	443	238	240	238
q12	389	208	212	208
q13	17780	2956	2931	2931
q14	236	216	206	206
q15	556	517	527	517
q16	487	417	395	395
q17	953	568	576	568
q18	7098	6482	6512	6482
q19	1866	961	907	907
q20	558	282	259	259
q21	3846	2971	3067	2971
q22	1085	973	984	973
Total cold run time: 109700 ms
Total hot run time: 39976 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7307	7160	7206	7160
q2	318	224	224	224
q3	2853	2836	2995	2836
q4	2002	1741	1825	1741
q5	5695	5704	5690	5690
q6	227	139	139	139
q7	2232	1773	1755	1755
q8	3341	3472	3531	3472
q9	8639	8785	8761	8761
q10	3518	3495	3493	3493
q11	577	488	485	485
q12	798	593	621	593
q13	16476	3076	3113	3076
q14	302	279	275	275
q15	565	527	515	515
q16	510	460	483	460
q17	1858	1649	1611	1611
q18	8247	7681	7683	7681
q19	6256	1398	1519	1398
q20	2115	1866	1819	1819
q21	5417	5216	5172	5172
q22	1114	1015	1030	1015
Total cold run time: 80367 ms
Total hot run time: 59371 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192588 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0841cee76005762132b4c8df3e6060a28014ffa9, data reload: false

query1	1251	873	849	849
query2	6269	1950	1983	1950
query3	10825	3833	3778	3778
query4	61243	26273	23391	23391
query5	5035	476	446	446
query6	373	167	161	161
query7	5307	315	292	292
query8	262	204	200	200
query9	7682	2666	2649	2649
query10	416	279	274	274
query11	15928	15347	15294	15294
query12	153	102	100	100
query13	1352	479	453	453
query14	10410	6733	7125	6733
query15	210	174	178	174
query16	6743	519	519	519
query17	1093	566	583	566
query18	1205	315	341	315
query19	209	162	147	147
query20	121	110	108	108
query21	207	110	101	101
query22	4703	4571	4390	4390
query23	34726	33792	33904	33792
query24	5956	2830	2779	2779
query25	497	391	396	391
query26	618	158	156	156
query27	1580	293	294	293
query28	4007	2210	2186	2186
query29	653	422	430	422
query30	223	153	157	153
query31	979	789	812	789
query32	67	50	55	50
query33	443	310	291	291
query34	872	481	503	481
query35	856	721	709	709
query36	1062	938	919	919
query37	130	71	79	71
query38	3928	3831	3913	3831
query39	1485	1421	1404	1404
query40	207	96	98	96
query41	44	43	42	42
query42	114	98	96	96
query43	509	477	480	477
query44	1111	760	756	756
query45	203	162	164	162
query46	1147	707	695	695
query47	1933	1841	1870	1841
query48	423	332	326	326
query49	681	388	395	388
query50	798	416	398	398
query51	7036	6948	6903	6903
query52	101	92	90	90
query53	253	187	174	174
query54	558	458	448	448
query55	74	75	72	72
query56	264	253	241	241
query57	1235	1132	1153	1132
query58	209	223	248	223
query59	3207	2855	2731	2731
query60	276	272	265	265
query61	99	101	97	97
query62	757	638	658	638
query63	209	183	171	171
query64	1236	618	644	618
query65	3252	3181	3131	3131
query66	678	299	304	299
query67	15717	15282	15170	15170
query68	3397	538	528	528
query69	704	278	290	278
query70	1166	1110	1132	1110
query71	472	267	269	267
query72	8197	3876	3885	3876
query73	755	330	333	330
query74	9713	8943	8851	8851
query75	4619	2632	2647	2632
query76	3833	905	893	893
query77	780	280	281	280
query78	9992	9189	9089	9089
query79	3387	563	589	563
query80	2456	423	426	423
query81	570	219	222	219
query82	1101	129	127	127
query83	315	130	136	130
query84	302	74	79	74
query85	1289	291	273	273
query86	413	296	271	271
query87	4354	4305	4157	4157
query88	4802	2399	2410	2399
query89	401	284	283	283
query90	2254	184	182	182
query91	134	106	106	106
query92	62	47	47	47
query93	5459	525	517	517
query94	1155	283	290	283
query95	346	242	247	242
query96	629	279	278	278
query97	3316	3169	3093	3093
query98	237	200	197	197
query99	1673	1303	1318	1303
Total cold run time: 319693 ms
Total hot run time: 192588 ms

@Gabriel39 Gabriel39 merged commit 8260737 into apache:branch-3.0 Oct 11, 2024
16 of 24 checks passed
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 11, 2024
Gabriel39 added a commit that referenced this pull request Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants