Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature](load) enable memtable on sink node by default #30372

Merged
merged 1 commit into from
Jan 25, 2024

Conversation

kaijchen
Copy link
Contributor

@kaijchen kaijchen commented Jan 25, 2024

Proposed changes

Set session variable enable_memtable_on_sink_node to true by default.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@kaijchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39466 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2b0da9a88e174f3e4f28bc94e826a9ab700e7a9f, data reload: false

------ Round 1 ----------------------------------
q1	17600	5332	5299	5299
q2	2040	139	134	134
q3	10827	1180	1183	1180
q4	10379	822	809	809
q5	8108	3236	3285	3236
q6	204	122	124	122
q7	887	516	507	507
q8	9550	1963	1986	1963
q9	7649	6556	6471	6471
q10	8223	3095	3037	3037
q11	414	230	208	208
q12	355	189	191	189
q13	17979	3388	3345	3345
q14	247	220	215	215
q15	546	506	501	501
q16	439	377	362	362
q17	942	597	532	532
q18	7750	7055	6904	6904
q19	1565	1395	1384	1384
q20	561	316	306	306
q21	2787	2474	2459	2459
q22	348	303	313	303
Total cold run time: 109400 ms
Total hot run time: 39466 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5386	5337	5220	5220
q2	323	217	207	207
q3	3365	3211	3210	3210
q4	2079	2061	2047	2047
q5	6126	5999	5996	5996
q6	197	112	118	112
q7	2302	1924	1898	1898
q8	3232	3397	3404	3397
q9	8900	8785	8829	8785
q10	3839	3840	3844	3840
q11	544	474	449	449
q12	822	645	607	607
q13	16896	3151	3156	3151
q14	269	254	260	254
q15	545	503	490	490
q16	513	454	461	454
q17	1844	1894	1835	1835
q18	9620	19022	9762	9762
q19	24207	1581	1494	1494
q20	4618	1954	1938	1938
q21	14462	5341	5389	5341
q22	956	525	514	514
Total cold run time: 111045 ms
Total hot run time: 61001 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185920 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b0da9a88e174f3e4f28bc94e826a9ab700e7a9f, data reload: false

query1	938	333	331	331
query2	6552	2003	1856	1856
query3	6691	205	199	199
query4	32416	22227	22062	22062
query5	4455	373	386	373
query6	239	154	164	154
query7	4597	253	251	251
query8	222	188	170	170
query9	8402	2616	2573	2573
query10	414	233	221	221
query11	16829	15618	15402	15402
query12	116	76	65	65
query13	1679	377	377	377
query14	10567	6985	6978	6978
query15	207	183	185	183
query16	5796	254	251	251
query17	951	462	479	462
query18	1791	257	244	244
query19	176	129	130	129
query20	64	68	68	68
query21	192	131	130	130
query22	4968	4858	4698	4698
query23	31567	30558	30803	30558
query24	12647	2819	2800	2800
query25	560	301	309	301
query26	1783	147	135	135
query27	3228	282	288	282
query28	7464	1824	1822	1822
query29	1983	623	627	623
query30	286	136	134	134
query31	925	729	778	729
query32	72	52	45	45
query33	695	207	214	207
query34	1161	461	459	459
query35	846	729	806	729
query36	1312	1220	1200	1200
query37	93	60	60	60
query38	3360	3259	3257	3257
query39	1315	1269	1238	1238
query40	341	86	83	83
query41	37	35	33	33
query42	86	77	80	77
query43	495	471	470	470
query44	1127	689	681	681
query45	190	174	170	170
query46	1063	652	640	640
query47	1610	1581	1562	1562
query48	385	307	311	307
query49	1191	290	281	281
query50	690	305	319	305
query51	5324	5211	5223	5211
query52	88	70	78	70
query53	313	253	249	249
query54	241	183	185	183
query55	79	75	71	71
query56	170	164	158	158
query57	1022	939	919	919
query58	183	152	152	152
query59	2903	2746	2668	2668
query60	204	173	175	173
query61	83	81	79	79
query62	588	356	356	356
query63	280	260	255	255
query64	6126	1786	1715	1715
query65	3331	3249	3245	3245
query66	1344	321	317	317
query67	15538	14986	14957	14957
query68	13157	534	516	516
query69	607	297	309	297
query70	1802	1524	1467	1467
query71	10473	10194	10192	10192
query72	4834	2818	2823	2818
query73	3219	315	316	315
query74	7028	6454	6397	6397
query75	5267	2311	2330	2311
query76	6684	1016	993	993
query77	687	230	219	219
query78	9047	8861	8742	8742
query79	1088	504	485	485
query80	546	319	308	308
query81	447	202	215	202
query82	202	77	83	77
query83	137	116	113	113
query84	273	69	67	67
query85	1077	322	315	315
query86	394	392	414	392
query87	3481	3322	3323	3322
query88	2933	2194	2174	2174
query89	441	349	347	347
query90	1989	183	179	179
query91	150	127	120	120
query92	49	43	41	41
query93	1552	451	413	413
query94	1235	154	154	154
query95	497	468	443	443
query96	632	325	313	313
query97	4221	4160	4199	4160
query98	193	194	192	192
query99	1014	693	726	693
Total cold run time: 307947 ms
Total hot run time: 185920 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.31 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2b0da9a88e174f3e4f28bc94e826a9ab700e7a9f, data reload: false

query1	0.03	0.02	0.02
query2	0.05	0.02	0.02
query3	0.22	0.05	0.05
query4	1.70	0.07	0.07
query5	0.54	0.53	0.52
query6	1.25	0.62	0.63
query7	0.02	0.02	0.01
query8	0.03	0.02	0.03
query9	0.54	0.49	0.50
query10	0.56	0.57	0.56
query11	0.11	0.08	0.08
query12	0.11	0.09	0.09
query13	0.61	0.60	0.60
query14	0.78	0.80	0.80
query15	0.78	0.78	0.78
query16	0.40	0.36	0.37
query17	1.01	1.02	1.01
query18	0.24	0.24	0.24
query19	1.89	1.83	1.82
query20	0.01	0.02	0.02
query21	15.42	0.60	0.56
query22	2.89	2.33	1.79
query23	17.33	0.79	0.72
query24	2.77	1.07	0.63
query25	0.39	0.11	0.05
query26	0.54	0.13	0.13
query27	0.07	0.06	0.05
query28	11.97	0.78	0.76
query29	12.49	3.09	3.16
query30	0.51	0.50	0.48
query31	2.78	0.35	0.34
query32	3.38	0.48	0.48
query33	3.17	3.25	3.22
query34	15.78	4.33	4.23
query35	4.29	4.31	4.28
query36	1.11	1.07	1.07
query37	0.06	0.04	0.05
query38	0.03	0.03	0.03
query39	0.02	0.02	0.01
query40	0.15	0.12	0.13
query41	0.07	0.01	0.02
query42	0.02	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 106.14 s
Total hot run time: 30.31 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 2b0da9a88e174f3e4f28bc94e826a9ab700e7a9f with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.0 seconds inserted 10000000 Rows, about 714K ops/s

Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 25, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 9cde5f7 into apache:master Jan 25, 2024
29 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants