Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[config](move-memtable) support HTTP header load_stream_per_node in stream load #34444

Merged
merged 4 commits into from
May 8, 2024
Merged

[config](move-memtable) support HTTP header load_stream_per_node in stream load #34444

merged 4 commits into from
May 8, 2024

Conversation

kaijchen
Copy link
Contributor

@kaijchen kaijchen commented May 6, 2024

Proposed changes

Support customize load_stream_per_node in HTTP header for stream load.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 6, 2024

clang-tidy review says "All clean, LGTM! 👍"

@kaijchen
Copy link
Contributor Author

kaijchen commented May 6, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 6, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.63% (8968/25172)
Line Coverage: 27.30% (74018/271157)
Region Coverage: 26.52% (38238/144164)
Branch Coverage: 23.33% (19487/83540)
Coverage Report: http://coverage.selectdb-in.cc/coverage/be8e1a780e82d45cb50f2bc35bef87b7f97fa307_be8e1a780e82d45cb50f2bc35bef87b7f97fa307/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 186905 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be8e1a780e82d45cb50f2bc35bef87b7f97fa307, data reload: false

query1	908	352	335	335
query2	6267	2398	2297	2297
query3	6652	204	212	204
query4	24208	22031	21986	21986
query5	3706	437	415	415
query6	246	195	186	186
query7	4532	309	297	297
query8	257	193	187	187
query9	8556	2421	2427	2421
query10	422	244	259	244
query11	15213	14849	14893	14849
query12	120	91	88	88
query13	1618	385	379	379
query14	10001	8336	6835	6835
query15	258	173	172	172
query16	8182	265	253	253
query17	1902	584	542	542
query18	2104	292	270	270
query19	339	158	150	150
query20	91	83	86	83
query21	195	126	135	126
query22	4974	4850	4811	4811
query23	34215	33277	33278	33277
query24	10716	2898	2912	2898
query25	601	380	383	380
query26	1168	154	150	150
query27	2337	311	330	311
query28	6995	2036	2053	2036
query29	873	609	656	609
query30	246	149	148	148
query31	921	744	718	718
query32	87	56	56	56
query33	744	252	254	252
query34	1007	467	467	467
query35	810	666	663	663
query36	1040	911	857	857
query37	130	64	64	64
query38	3175	3015	2989	2989
query39	1611	1532	1528	1528
query40	197	124	131	124
query41	40	38	38	38
query42	101	95	95	95
query43	590	523	552	523
query44	1217	724	723	723
query45	271	259	270	259
query46	1084	763	712	712
query47	1956	1858	1859	1858
query48	360	301	296	296
query49	865	379	385	379
query50	761	399	388	388
query51	6820	6686	6586	6586
query52	101	86	91	86
query53	356	278	285	278
query54	321	258	233	233
query55	82	76	74	74
query56	244	228	213	213
query57	1185	1114	1142	1114
query58	230	196	198	196
query59	3378	3240	3248	3240
query60	245	225	227	225
query61	88	85	86	85
query62	638	465	448	448
query63	313	275	281	275
query64	8641	7245	7182	7182
query65	3110	3050	3042	3042
query66	826	340	333	333
query67	15733	14979	14868	14868
query68	9872	538	529	529
query69	611	306	308	306
query70	1355	1105	1085	1085
query71	553	267	271	267
query72	8652	2508	2416	2416
query73	1583	325	319	319
query74	6525	6117	6088	6088
query75	5620	2666	2627	2627
query76	5962	1010	913	913
query77	802	262	260	260
query78	11145	10244	10239	10239
query79	11792	531	537	531
query80	1745	441	431	431
query81	497	221	220	220
query82	241	91	94	91
query83	225	167	167	167
query84	272	86	84	84
query85	1056	275	273	273
query86	341	294	307	294
query87	3269	3101	3055	3055
query88	5361	2358	2346	2346
query89	500	374	372	372
query90	2257	181	184	181
query91	127	97	100	97
query92	60	49	48	48
query93	7201	508	500	500
query94	1628	188	185	185
query95	402	308	307	307
query96	611	268	263	263
query97	3139	2934	2924	2924
query98	236	219	218	218
query99	1213	866	863	863
Total cold run time: 312033 ms
Total hot run time: 186905 ms

liaoxin01
liaoxin01 approved these changes May 7, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kaijchen
Copy link
Contributor Author

kaijchen commented May 7, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 7, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 41180 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 18c8974ae3754ceb1800ab28d86a2e8470b5f6e1, data reload: false

------ Round 1 ----------------------------------
q1	17587	4266	4225	4225
q2	2007	189	193	189
q3	10467	1230	1161	1161
q4	10197	828	734	734
q5	7486	2655	2736	2655
q6	214	130	135	130
q7	1038	607	610	607
q8	9239	2169	2103	2103
q9	9545	6807	6816	6807
q10	9568	3897	3904	3897
q11	449	246	244	244
q12	511	219	218	218
q13	17549	3169	3203	3169
q14	263	221	211	211
q15	527	477	464	464
q16	512	428	395	395
q17	975	708	712	708
q18	8166	7765	7796	7765
q19	2279	1511	1524	1511
q20	639	328	319	319
q21	5373	4134	3387	3387
q22	354	288	281	281
Total cold run time: 114945 ms
Total hot run time: 41180 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4474	4405	4377	4377
q2	386	272	279	272
q3	3185	2942	2954	2942
q4	1911	1596	1615	1596
q5	5442	5528	5472	5472
q6	223	126	165	126
q7	2333	1979	1967	1967
q8	3300	3407	3355	3355
q9	8718	8675	8748	8675
q10	3979	3714	3875	3714
q11	600	479	491	479
q12	821	631	645	631
q13	17153	3143	3202	3143
q14	296	291	264	264
q15	513	483	502	483
q16	491	422	426	422
q17	1733	1484	1443	1443
q18	7625	7626	7524	7524
q19	1669	1529	1571	1529
q20	1969	1752	1779	1752
q21	5003	5014	4862	4862
q22	552	503	481	481
Total cold run time: 72376 ms
Total hot run time: 55509 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185627 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 18c8974ae3754ceb1800ab28d86a2e8470b5f6e1, data reload: false

query1	898	365	345	345
query2	6447	2349	2308	2308
query3	6648	218	223	218
query4	23653	21270	21189	21189
query5	4210	421	417	417
query6	275	186	173	173
query7	4595	284	285	284
query8	247	190	190	190
query9	8498	2388	2389	2388
query10	427	248	234	234
query11	14833	14174	14337	14174
query12	135	93	99	93
query13	1645	357	367	357
query14	8722	8549	6648	6648
query15	219	175	167	167
query16	7132	255	255	255
query17	1032	571	561	561
query18	1927	289	295	289
query19	209	154	160	154
query20	95	86	91	86
query21	196	126	131	126
query22	5164	4953	4910	4910
query23	33705	33329	33238	33238
query24	5110	3002	2879	2879
query25	515	395	382	382
query26	692	155	153	153
query27	1849	312	321	312
query28	3780	2050	2041	2041
query29	836	626	617	617
query30	239	161	159	159
query31	919	758	714	714
query32	63	54	55	54
query33	391	262	252	252
query34	871	481	478	478
query35	765	660	660	660
query36	1041	890	891	890
query37	109	67	70	67
query38	3134	3066	2980	2980
query39	1614	1531	1528	1528
query40	199	132	132	132
query41	45	43	41	41
query42	109	100	95	95
query43	551	522	549	522
query44	1061	728	746	728
query45	275	255	255	255
query46	1055	708	701	701
query47	1963	1900	1867	1867
query48	368	307	293	293
query49	779	408	432	408
query50	782	386	383	383
query51	6761	6625	6699	6625
query52	109	95	90	90
query53	344	282	283	282
query54	272	252	256	252
query55	77	73	76	73
query56	249	228	225	225
query57	1242	1147	1158	1147
query58	222	201	203	201
query59	3636	3189	3125	3125
query60	247	232	242	232
query61	93	88	88	88
query62	541	466	452	452
query63	302	286	280	280
query64	7542	7230	7151	7151
query65	3107	3029	3044	3029
query66	665	334	335	334
query67	15798	15523	14956	14956
query68	9774	551	523	523
query69	602	309	309	309
query70	1147	1084	1108	1084
query71	557	308	272	272
query72	8290	2549	2420	2420
query73	1177	334	326	326
query74	6592	6210	6271	6210
query75	4723	2643	2604	2604
query76	5196	972	1089	972
query77	692	268	265	265
query78	10939	10285	10343	10285
query79	11928	508	516	508
query80	2195	448	434	434
query81	497	229	220	220
query82	236	92	98	92
query83	201	168	162	162
query84	264	84	86	84
query85	863	281	267	267
query86	343	312	312	312
query87	3451	3072	3093	3072
query88	5460	2323	2325	2323
query89	498	391	384	384
query90	2231	183	176	176
query91	122	98	96	96
query92	62	47	47	47
query93	6564	519	498	498
query94	1217	193	178	178
query95	392	301	302	301
query96	613	273	264	264
query97	3170	3001	2937	2937
query98	230	218	215	215
query99	1171	873	850	850
Total cold run time: 292631 ms
Total hot run time: 185627 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.64% (8970/25167)
Line Coverage: 27.28% (74021/271300)
Region Coverage: 26.51% (38241/144248)
Branch Coverage: 23.31% (19491/83618)
Coverage Report: http://coverage.selectdb-in.cc/coverage/18c8974ae3754ceb1800ab28d86a2e8470b5f6e1_18c8974ae3754ceb1800ab28d86a2e8470b5f6e1/report/index.html

@kaijchen
Copy link
Contributor Author

kaijchen commented May 7, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 7, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.68% (8981/25168)
Line Coverage: 27.35% (74210/271332)
Region Coverage: 26.60% (38367/144231)
Branch Coverage: 23.41% (19566/83596)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b1b409a79a33f5f38a1ff7b0601cdbd7840a93d2_b1b409a79a33f5f38a1ff7b0601cdbd7840a93d2/report/index.html

dataroaring
dataroaring approved these changes May 8, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 8, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 8, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 8, 2024

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit b78830d into apache:master May 8, 2024
23 of 26 checks passed
yiguolei pushed a commit that referenced this pull request May 10, 2024
@kaijchen @Doris-Extras
[config](move-memtable) support HTTP header load_stream_per_node in s…
95593f8 
…tream load (#34444)
ByteYue pushed a commit to ByteYue/doris that referenced this pull request May 15, 2024
@kaijchen @ByteYue
[config](move-memtable) support HTTP header load_stream_per_node in s…
32a28ad 
…tream load (apache#34444)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@liaoxin01 liaoxin01 liaoxin01 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kaijchen @doris-robot @liaoxin01 @dataroaring