[fix](move-memtable) all sinks wait stream close for load timeout #30356

kaijchen · 2024-01-25T05:11:21Z

Proposed changes

Reverts commit ae02202.
Also set stream close wait to load timeout.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

…che#30098)" This reverts commit ae02202.

kaijchen · 2024-01-25T05:11:30Z

run buildall

github-actions · 2024-01-25T05:20:50Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-01-25T06:13:13Z

TPC-H: Total hot run time: 39285 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dd1c768161397c727998f6467cf85c412f699940, data reload: false

------ Round 1 ----------------------------------
q1	18013	5787	5101	5101
q2	2044	136	135	135
q3	11192	1179	1171	1171
q4	10577	757	787	757
q5	8302	3238	3210	3210
q6	200	125	125	125
q7	896	512	489	489
q8	9523	2035	2021	2021
q9	7978	6587	6514	6514
q10	8282	3038	3084	3038
q11	405	206	218	206
q12	358	193	191	191
q13	17996	3382	3375	3375
q14	253	219	224	219
q15	556	514	506	506
q16	424	383	396	383
q17	945	512	517	512
q18	7508	7069	6948	6948
q19	1696	1339	1378	1339
q20	579	297	301	297
q21	2777	2450	2436	2436
q22	367	313	312	312
Total cold run time: 110871 ms
Total hot run time: 39285 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5495	5180	5350	5180
q2	332	216	211	211
q3	3346	3222	3171	3171
q4	2074	2040	2026	2026
q5	6039	5838	5907	5838
q6	198	118	117	117
q7	2297	1806	1853	1806
q8	3241	3372	3408	3372
q9	8932	8948	8750	8750
q10	3967	3771	3823	3771
q11	558	480	456	456
q12	781	619	614	614
q13	16940	3205	3174	3174
q14	281	264	267	264
q15	551	505	515	505
q16	499	487	470	470
q17	2798	1864	1809	1809
q18	9461	9538	9155	9155
q19	19276	1538	1546	1538
q20	4605	1937	1922	1922
q21	14176	5273	5290	5273
q22	964	562	542	542
Total cold run time: 106811 ms
Total hot run time: 59964 ms

dataroaring

LGTM

github-actions · 2024-01-25T06:16:41Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-01-25T06:16:44Z

PR approved by anyone and no changes requested.

doris-robot · 2024-01-25T06:25:14Z

TPC-DS: Total hot run time: 186633 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dd1c768161397c727998f6467cf85c412f699940, data reload: false

query1	925	339	334	334
query2	6571	1894	1967	1894
query3	6751	198	192	192
query4	33025	22087	22214	22087
query5	4448	382	430	382
query6	250	168	150	150
query7	4601	264	268	264
query8	221	187	180	180
query9	8367	2588	2587	2587
query10	439	220	219	219
query11	16641	15608	15551	15551
query12	123	67	67	67
query13	1676	369	375	369
query14	10527	7013	7058	7013
query15	215	188	187	187
query16	5795	257	245	245
query17	953	468	465	465
query18	1793	256	252	252
query19	178	135	132	132
query20	74	77	63	63
query21	182	122	137	122
query22	5013	4825	4734	4734
query23	31569	30776	30819	30776
query24	11534	2845	2798	2798
query25	572	309	318	309
query26	1599	142	145	142
query27	3230	285	285	285
query28	7104	1841	1832	1832
query29	1543	640	620	620
query30	275	139	136	136
query31	928	740	738	738
query32	72	53	48	48
query33	706	219	212	212
query34	1121	452	465	452
query35	870	787	770	770
query36	1397	1201	1199	1199
query37	90	58	57	57
query38	3352	3245	3238	3238
query39	1316	1282	1258	1258
query40	199	85	84	84
query41	37	35	35	35
query42	93	83	80	80
query43	529	479	484	479
query44	1067	690	691	690
query45	190	176	173	173
query46	1071	660	647	647
query47	1666	1612	1499	1499
query48	391	310	309	309
query49	1130	294	295	294
query50	697	312	313	312
query51	5334	5210	5168	5168
query52	90	79	78	78
query53	321	253	261	253
query54	245	182	178	178
query55	76	80	82	80
query56	174	160	162	160
query57	1000	916	919	916
query58	180	154	182	154
query59	2909	2814	2792	2792
query60	200	184	188	184
query61	80	80	80	80
query62	596	351	365	351
query63	287	265	259	259
query64	5095	1770	1735	1735
query65	3343	3262	3252	3252
query66	1296	315	304	304
query67	15627	15174	15049	15049
query68	12835	524	519	519
query69	595	290	285	285
query70	1736	1529	1549	1529
query71	10430	10202	10198	10198
query72	4890	2807	2806	2806
query73	3176	322	304	304
query74	6857	6430	6367	6367
query75	4923	2336	2320	2320
query76	6315	1046	1031	1031
query77	663	223	225	223
query78	8884	8798	8618	8618
query79	3389	493	504	493
query80	557	318	319	318
query81	482	200	202	200
query82	233	76	85	76
query83	136	117	116	116
query84	268	65	68	65
query85	1050	332	319	319
query86	409	387	381	381
query87	3559	3321	3346	3321
query88	3260	2171	2153	2153
query89	445	356	358	356
query90	1950	188	183	183
query91	149	117	128	117
query92	50	44	48	44
query93	2030	449	408	408
query94	1256	163	157	157
query95	502	468	456	456
query96	618	307	325	307
query97	4274	4117	4173	4117
query98	204	185	187	185
query99	1024	728	707	707
Total cold run time: 307118 ms
Total hot run time: 186633 ms

doris-robot · 2024-01-25T06:30:01Z

TeamCity be ut coverage result:
Function Coverage: 36.26% (8610/23745)
Line Coverage: 28.37% (70462/248403)
Region Coverage: 27.34% (36350/132974)
Branch Coverage: 24.14% (18635/77192)
Coverage Report: http://coverage.selectdb-in.cc/coverage/dd1c768161397c727998f6467cf85c412f699940_dd1c768161397c727998f6467cf85c412f699940/report/index.html

doris-robot · 2024-01-25T06:30:39Z

ClickBench: Total hot run time: 29.77 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dd1c768161397c727998f6467cf85c412f699940, data reload: false

query1	0.03	0.03	0.02
query2	0.06	0.02	0.03
query3	0.23	0.05	0.04
query4	1.71	0.07	0.07
query5	0.54	0.52	0.53
query6	1.31	0.63	0.64
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.55	0.49	0.49
query10	0.55	0.56	0.57
query11	0.12	0.09	0.09
query12	0.12	0.09	0.08
query13	0.60	0.61	0.60
query14	0.78	0.80	0.80
query15	0.80	0.79	0.77
query16	0.40	0.38	0.37
query17	1.05	1.02	1.04
query18	0.24	0.25	0.25
query19	1.81	1.80	1.79
query20	0.01	0.01	0.01
query21	15.42	0.58	0.60
query22	2.64	2.73	0.78
query23	17.16	0.84	0.86
query24	2.30	0.69	1.53
query25	0.42	0.25	0.15
query26	0.49	0.13	0.12
query27	0.06	0.05	0.05
query28	11.33	0.75	0.79
query29	12.51	3.23	3.24
query30	0.56	0.51	0.49
query31	2.78	0.34	0.35
query32	3.37	0.49	0.48
query33	3.26	3.23	3.24
query34	15.76	4.23	4.23
query35	4.25	4.31	4.35
query36	1.10	1.08	1.08
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.02
query40	0.17	0.13	0.13
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 104.79 s
Total hot run time: 29.77 s

doris-robot · 2024-01-25T06:33:43Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit dd1c768161397c727998f6467cf85c412f699940 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.7 seconds inserted 10000000 Rows, about 680K ops/s

github-actions · 2024-01-25T06:46:04Z

clang-tidy review says "All clean, LGTM! 👍"

kaijchen · 2024-01-25T07:00:01Z

run buildall

dataroaring

LGTM

github-actions · 2024-01-25T07:26:49Z

PR approved by at least one committer and no changes requested.

doris-robot · 2024-01-25T07:27:51Z

TPC-H: Total hot run time: 38789 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 472932611f8a805d13333d5b27b8f191456bdd38, data reload: false

------ Round 1 ----------------------------------
q1	17732	5542	5339	5339
q2	2040	141	127	127
q3	10816	1163	1177	1163
q4	10434	762	809	762
q5	7747	3146	3116	3116
q6	200	117	121	117
q7	841	488	481	481
q8	9212	1882	1933	1882
q9	7235	6380	6304	6304
q10	8179	3056	3065	3056
q11	403	224	213	213
q12	353	187	187	187
q13	17978	3394	3348	3348
q14	241	216	220	216
q15	537	507	506	506
q16	431	360	385	360
q17	932	502	528	502
q18	7400	6938	6835	6835
q19	1870	1311	1376	1311
q20	602	294	305	294
q21	2756	2370	2418	2370
q22	355	309	300	300
Total cold run time: 108294 ms
Total hot run time: 38789 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5513	5346	5249	5249
q2	329	217	211	211
q3	3339	3195	3214	3195
q4	2109	2040	2023	2023
q5	5827	5894	5789	5789
q6	198	118	117	117
q7	2289	1871	1832	1832
q8	3219	3362	3372	3362
q9	9007	8885	8875	8875
q10	3988	3842	3795	3795
q11	560	447	456	447
q12	804	576	634	576
q13	16936	3157	3161	3157
q14	275	254	248	248
q15	557	492	506	492
q16	510	462	460	460
q17	4174	1846	1877	1846
q18	9493	9517	9091	9091
q19	19141	1542	1533	1533
q20	4576	1940	1917	1917
q21	14147	5358	5403	5358
q22	958	570	569	569
Total cold run time: 107949 ms
Total hot run time: 60142 ms

doris-robot · 2024-01-25T07:40:01Z

TPC-DS: Total hot run time: 187306 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 472932611f8a805d13333d5b27b8f191456bdd38, data reload: false

query1	929	333	332	332
query2	6547	2229	1946	1946
query3	6694	198	194	194
query4	31113	22364	22461	22364
query5	4551	434	358	358
query6	234	152	161	152
query7	4615	256	264	256
query8	223	173	181	173
query9	8337	2588	2563	2563
query10	420	243	223	223
query11	17444	15652	15592	15592
query12	110	69	65	65
query13	1671	389	388	388
query14	10544	6986	7024	6986
query15	211	178	187	178
query16	5797	254	243	243
query17	958	477	481	477
query18	1786	257	247	247
query19	173	135	135	135
query20	70	67	70	67
query21	193	134	131	131
query22	4970	5120	5129	5120
query23	32614	31287	31123	31123
query24	12764	2806	2797	2797
query25	559	311	309	309
query26	1828	146	143	143
query27	3626	280	281	280
query28	7510	1851	1830	1830
query29	2183	634	631	631
query30	290	134	143	134
query31	939	726	722	722
query32	78	52	47	47
query33	712	221	209	209
query34	1311	463	459	459
query35	862	779	749	749
query36	1319	1149	1209	1149
query37	176	55	58	55
query38	3382	3261	3206	3206
query39	1319	1259	1258	1258
query40	342	86	78	78
query41	36	35	35	35
query42	89	80	78	78
query43	520	485	482	482
query44	1070	678	671	671
query45	194	181	175	175
query46	1068	648	658	648
query47	1620	1559	1593	1559
query48	377	300	302	300
query49	1221	291	288	288
query50	697	307	306	306
query51	5268	5239	5205	5205
query52	85	76	72	72
query53	325	253	256	253
query54	245	182	192	182
query55	85	73	76	73
query56	177	180	157	157
query57	987	941	930	930
query58	187	154	159	154
query59	2903	2794	2714	2714
query60	216	177	187	177
query61	81	86	80	80
query62	601	378	375	375
query63	282	255	260	255
query64	6098	1747	1765	1747
query65	3319	3253	3227	3227
query66	1421	321	306	306
query67	15631	15075	14912	14912
query68	12216	508	501	501
query69	608	295	286	286
query70	1726	1519	1445	1445
query71	10471	10202	10200	10200
query72	4945	2811	2824	2811
query73	2713	314	317	314
query74	6949	6328	6373	6328
query75	5167	2320	2296	2296
query76	6712	1057	1028	1028
query77	636	224	236	224
query78	9056	8887	8638	8638
query79	4648	487	487	487
query80	698	332	315	315
query81	447	203	197	197
query82	268	79	78	78
query83	135	109	115	109
query84	271	66	70	66
query85	1060	338	328	328
query86	404	388	382	382
query87	3486	3328	3333	3328
query88	3379	2135	2145	2135
query89	446	370	356	356
query90	1995	191	184	184
query91	149	127	121	121
query92	55	43	43	43
query93	3033	417	422	417
query94	1264	163	152	152
query95	510	453	451	451
query96	629	315	309	309
query97	4267	4149	4171	4149
query98	209	191	182	182
query99	1024	716	758	716
Total cold run time: 313783 ms
Total hot run time: 187306 ms

doris-robot · 2024-01-25T07:45:27Z

ClickBench: Total hot run time: 30.59 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 472932611f8a805d13333d5b27b8f191456bdd38, data reload: false

query1	0.03	0.02	0.02
query2	0.06	0.02	0.02
query3	0.22	0.05	0.05
query4	1.69	0.06	0.07
query5	0.54	0.52	0.52
query6	1.31	0.63	0.63
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.55	0.52	0.50
query10	0.56	0.56	0.56
query11	0.11	0.10	0.09
query12	0.11	0.09	0.09
query13	0.61	0.61	0.60
query14	0.78	0.83	0.78
query15	0.78	0.77	0.78
query16	0.38	0.38	0.37
query17	1.03	1.01	1.04
query18	0.22	0.27	0.21
query19	1.88	1.75	1.78
query20	0.02	0.01	0.01
query21	15.40	0.57	0.57
query22	2.70	2.19	2.10
query23	17.09	0.96	0.71
query24	2.61	0.52	1.86
query25	0.40	0.23	0.19
query26	0.69	0.13	0.13
query27	0.06	0.04	0.06
query28	11.32	0.77	0.77
query29	12.55	3.09	3.18
query30	0.54	0.50	0.49
query31	2.82	0.33	0.34
query32	3.40	0.49	0.48
query33	3.20	3.25	3.25
query34	16.04	4.26	4.22
query35	4.33	4.31	4.30
query36	1.10	1.08	1.07
query37	0.07	0.05	0.05
query38	0.04	0.03	0.02
query39	0.02	0.01	0.01
query40	0.16	0.13	0.13
query41	0.07	0.02	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.59 s
Total hot run time: 30.59 s

doris-robot · 2024-01-25T07:48:32Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 472932611f8a805d13333d5b27b8f191456bdd38 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.2 seconds inserted 10000000 Rows, about 704K ops/s

sollhui

LGTM

…0356)

kaijchen added 2 commits January 25, 2024 13:02

Revert "[fix](move-memtable) only do close wait on the last sink (apa…

23b9e91

…che#30098)" This reverts commit ae02202.

[fix](move-memtable) all sinks wait stream close for load timeout

dd1c768

dataroaring previously approved these changes Jan 25, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 25, 2024

github-actions bot added the reviewed label Jan 25, 2024

use ramain time as timeout

4729326

kaijchen dismissed dataroaring’s stale review via 4729326 January 25, 2024 06:35

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 25, 2024

dataroaring approved these changes Jan 25, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 25, 2024

liaoxin01 approved these changes Jan 25, 2024

View reviewed changes

sollhui approved these changes Jan 25, 2024

View reviewed changes

dataroaring merged commit 520293a into apache:master Jan 25, 2024
29 of 30 checks passed

yiguolei pushed a commit that referenced this pull request Jan 25, 2024

[fix](move-memtable) all sinks wait stream close for load timeout (#3…

90ed663

…0356)

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](move-memtable) all sinks wait stream close for load timeout #30356

[fix](move-memtable) all sinks wait stream close for load timeout #30356

kaijchen commented Jan 25, 2024

kaijchen commented Jan 25, 2024

github-actions bot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

dataroaring left a comment

github-actions bot commented Jan 25, 2024

github-actions bot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

github-actions bot commented Jan 25, 2024

kaijchen commented Jan 25, 2024

dataroaring left a comment

github-actions bot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

sollhui left a comment

[fix](move-memtable) all sinks wait stream close for load timeout #30356

[fix](move-memtable) all sinks wait stream close for load timeout #30356

Conversation

kaijchen commented Jan 25, 2024

Proposed changes

Further comments

kaijchen commented Jan 25, 2024

github-actions bot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 25, 2024

github-actions bot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

github-actions bot commented Jan 25, 2024

kaijchen commented Jan 25, 2024

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

doris-robot commented Jan 25, 2024

sollhui left a comment

Choose a reason for hiding this comment