Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](cloud-mow) MS should delete the existing keys before rewriting it when processing old version delete bitmap on cu compaction #43479

Merged
merged 1 commit into from
Nov 8, 2024

Conversation

hust-hhb
Copy link
Contributor

@hust-hhb hust-hhb commented Nov 8, 2024

pr #40204 support deleting old version delete bitmap when doing cu compaction, it will update delete bitmap with agg result, then deleting old version delete bitmap. Updating delete bitmap means overwriting existing keys, however delete bitmap may split into multiple kvs to store in fdb, so we should delete the existing keys and rewrite it to fdb instead of overwriting directly.
pick pr #42379

…it when processing old version delete bitmap on cu compaction
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Nov 8, 2024

run buildall

Copy link
Contributor

github-actions bot commented Nov 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40613 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 29ac1e7ee4cac43de85b1943750a68cd4b24076b, data reload: false

------ Round 1 ----------------------------------
q1	17577	7455	7331	7331
q2	2061	164	166	164
q3	10790	1063	1141	1063
q4	10544	783	741	741
q5	7748	2823	2739	2739
q6	234	150	145	145
q7	978	614	597	597
q8	9371	1967	2031	1967
q9	6636	6444	6404	6404
q10	7009	2269	2296	2269
q11	461	262	263	262
q12	407	217	213	213
q13	17776	2949	2950	2949
q14	251	213	207	207
q15	559	514	513	513
q16	657	626	591	591
q17	976	597	576	576
q18	7207	6723	6490	6490
q19	1385	1065	1077	1065
q20	482	195	190	190
q21	3894	3163	3262	3163
q22	1103	981	974	974
Total cold run time: 108106 ms
Total hot run time: 40613 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7358	7265	7233	7233
q2	408	313	300	300
q3	2898	2874	2895	2874
q4	1979	1755	1794	1755
q5	5699	5677	5715	5677
q6	225	140	149	140
q7	2187	1809	1766	1766
q8	3333	3573	3526	3526
q9	8819	8856	8818	8818
q10	3524	3500	3507	3500
q11	608	487	504	487
q12	809	624	597	597
q13	16510	3142	3093	3093
q14	298	274	275	274
q15	589	543	554	543
q16	710	665	665	665
q17	1853	1653	1617	1617
q18	8270	7753	7606	7606
q19	8371	1595	1461	1461
q20	2072	1865	1859	1859
q21	5372	5190	5361	5190
q22	1117	1002	999	999
Total cold run time: 83009 ms
Total hot run time: 59980 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192694 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 29ac1e7ee4cac43de85b1943750a68cd4b24076b, data reload: false

query1	2454	2253	2262	2253
query2	6215	2200	2013	2013
query3	14468	9934	248	248
query4	33559	23333	23350	23333
query5	4684	444	433	433
query6	286	177	183	177
query7	3983	302	313	302
query8	286	236	232	232
query9	9405	2697	2663	2663
query10	493	261	256	256
query11	18160	15011	15153	15011
query12	151	104	106	104
query13	1550	418	405	405
query14	9570	7151	6889	6889
query15	217	186	192	186
query16	7693	532	490	490
query17	1583	627	584	584
query18	2244	617	637	617
query19	230	180	190	180
query20	124	115	111	111
query21	209	104	115	104
query22	4873	4137	4138	4137
query23	34707	33865	34305	33865
query24	12554	3368	3299	3299
query25	683	404	401	401
query26	1768	184	185	184
query27	2651	292	301	292
query28	7891	2516	2470	2470
query29	1091	442	430	430
query30	390	302	303	302
query31	1055	790	806	790
query32	96	56	56	56
query33	770	277	282	277
query34	945	497	496	496
query35	900	717	729	717
query36	1105	931	949	931
query37	257	74	72	72
query38	3934	3836	3861	3836
query39	1474	1424	1459	1424
query40	255	97	99	97
query41	50	51	49	49
query42	106	97	95	95
query43	539	509	502	502
query44	1189	768	787	768
query45	181	166	169	166
query46	1132	764	718	718
query47	1898	1777	1789	1777
query48	463	374	361	361
query49	1083	401	407	401
query50	799	408	407	407
query51	7196	7108	6916	6916
query52	102	93	91	91
query53	256	187	182	182
query54	1314	445	456	445
query55	78	80	74	74
query56	241	233	223	223
query57	1157	1093	1086	1086
query58	223	197	223	197
query59	3058	2933	2889	2889
query60	275	245	244	244
query61	108	129	109	109
query62	849	654	667	654
query63	217	185	178	178
query64	5036	642	604	604
query65	3264	3178	3161	3161
query66	1398	306	300	300
query67	15719	15422	15206	15206
query68	5826	583	575	575
query69	422	250	263	250
query70	1204	1137	1132	1132
query71	453	260	254	254
query72	6174	3946	4096	3946
query73	761	341	338	338
query74	10465	8849	8960	8849
query75	3373	2639	2651	2639
query76	3304	1047	985	985
query77	442	282	286	282
query78	10603	9776	9538	9538
query79	8115	585	586	585
query80	2072	420	421	420
query81	582	240	239	239
query82	1438	110	114	110
query83	312	158	155	155
query84	292	73	74	73
query85	2160	316	289	289
query86	489	311	302	302
query87	4457	4277	4289	4277
query88	5688	2380	2379	2379
query89	559	305	282	282
query90	2251	181	182	181
query91	176	144	143	143
query92	61	49	48	48
query93	7820	539	516	516
query94	1038	284	280	280
query95	349	245	250	245
query96	628	277	285	277
query97	3422	3164	3155	3155
query98	220	194	192	192
query99	1864	1288	1311	1288
Total cold run time: 329914 ms
Total hot run time: 192694 ms

@zhannngchen zhannngchen merged commit 3bd553d into apache:branch-3.0 Nov 8, 2024
21 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants