[Fix](full compaction) Full compaction should not do ordered data compaction #44359

Yukang-Lian · 2024-11-20T12:29:54Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Problem: For a duplicate table with the following distribution, if it has already completed cumulative compaction and then undergoes full compaction, it will cause a BE core issue. Check failed: new_point == Tablet::K_INVALID_CUMULATIVE_POINT || new_point >= _cumulative_point Unexpected cumulative point: 1087, origin: 2801.
"rowsets": [
"[0-386] 0 DATA NONOVERLAPPING 02000000000198aabe4290f2b0f5f35610c08a233a061892 0",
"[387-387] 0 DELETE OVERLAP_UNKNOWN 0200000000541310ac4d76e7580a708a2823a4d7a4f06090 0",
"[388-388] 0 DELETE OVERLAP_UNKNOWN 0200000000541d76ac4d76e7580a708a2823a4d7a4f06090 0",
"[389-389] 0 DELETE OVERLAP_UNKNOWN 0200000000543b4dac4d76e7580a708a2823a4d7a4f06090 0",
"[390-390] 0 DELETE OVERLAP_UNKNOWN 02000000005453aeac4d76e7580a708a2823a4d7a4f06090 0",
"[391-391] 0 DELETE OVERLAP_UNKNOWN 0200000000546a44ac4d76e7580a708a2823a4d7a4f06090 0",
"[392-392] 0 DELETE OVERLAP_UNKNOWN 02000000005480dbac4d76e7580a708a2823a4d7a4f06090 0",
"[393-393] 0 DELETE OVERLAP_UNKNOWN 0200000000548cb3ac4d76e7580a708a2823a4d7a4f06090 0",
"[394-394] 0 DELETE OVERLAP_UNKNOWN 0200000000549a25ac4d76e7580a708a2823a4d7a4f06090 0",
"[395-395] 0 DELETE OVERLAP_UNKNOWN 020000000054b359ac4d76e7580a708a2823a4d7a4f06090 0",
"[396-396] 0 DELETE OVERLAP_UNKNOWN 020000000054c19dac4d76e7580a708a2823a4d7a4f06090 0",
"[397-397] 0 DELETE OVERLAP_UNKNOWN 020000000054d757ac4d76e7580a708a2823a4d7a4f06090 0",
...
"[1085-1085] 0 DELETE OVERLAP_UNKNOWN 02000000002a0b20bd4798638f237008ff42fbca276b52a2 0",
"[1087-1506] 1 DATA NONOVERLAPPING 020000000000047e3b452de14ceaad2e78a87526026d2290 326.10 KB",
"[1087-1506] 1 DATA NONOVERLAPPING 020000000000047e3b452de14ceaad2e78a87526026d2290 326.10 KB",
...
"[2800-2800] 0 DELETE OVERLAP_UNKNOWN 02000000002f12d6bd4798638f237008ff42fbca276b52a2 0"

Reason: The duplicate table will go through ordered data compaction. Due to the special distribution of the table, the input rowset will be cut by the ordered data compaction, resulting in the full compaction only being performed on a part of the rowsets.

Solution: For full compaction, prohibit ordered data compaction.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-20T12:29:59Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Yukang-Lian · 2024-11-20T12:30:05Z

run buildall

Yukang-Lian · 2024-11-20T12:30:38Z

regression test will be added later.

github-actions · 2024-11-20T12:34:42Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-11-20T12:48:56Z

TPC-H: Total hot run time: 40098 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 085de8b2f3b2136fc6a4b9745c25da01db2ef5cb, data reload: false

------ Round 1 ----------------------------------
q1	17619	7526	7390	7390
q2	2037	172	172	172
q3	10548	1148	1114	1114
q4	10217	812	782	782
q5	7586	2726	2710	2710
q6	241	153	150	150
q7	983	625	597	597
q8	9255	1848	1939	1848
q9	6492	6398	6419	6398
q10	6971	2283	2353	2283
q11	462	254	280	254
q12	412	225	214	214
q13	17758	3000	3040	3000
q14	245	215	224	215
q15	576	523	542	523
q16	648	585	591	585
q17	982	559	602	559
q18	7491	6674	6739	6674
q19	1326	959	992	959
q20	476	181	190	181
q21	3993	3287	3175	3175
q22	375	315	315	315
Total cold run time: 106693 ms
Total hot run time: 40098 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7384	7351	7334	7334
q2	336	234	229	229
q3	2904	2786	2789	2786
q4	1987	1762	1729	1729
q5	5415	5413	5422	5413
q6	223	143	143	143
q7	2109	1719	1686	1686
q8	3254	3421	3424	3421
q9	8535	8530	8555	8530
q10	3535	3530	3432	3432
q11	604	502	507	502
q12	812	573	604	573
q13	11608	3043	3061	3043
q14	306	273	263	263
q15	560	515	512	512
q16	684	636	631	631
q17	1807	1591	1577	1577
q18	7787	7570	7451	7451
q19	1671	1492	1516	1492
q20	2047	1839	1807	1807
q21	5566	5305	5189	5189
q22	627	544	571	544
Total cold run time: 69761 ms
Total hot run time: 58287 ms

doris-robot · 2024-11-20T13:01:04Z

TPC-DS: Total hot run time: 192019 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 085de8b2f3b2136fc6a4b9745c25da01db2ef5cb, data reload: false

query1	966	409	390	390
query2	6536	2111	2011	2011
query3	6714	217	217	217
query4	34014	23575	23547	23547
query5	4369	460	442	442
query6	284	191	186	186
query7	4641	307	309	307
query8	305	238	228	228
query9	9313	2659	2647	2647
query10	468	258	247	247
query11	18094	15337	15267	15267
query12	159	106	109	106
query13	1661	415	422	415
query14	10270	7330	7037	7037
query15	301	175	178	175
query16	8087	470	466	466
query17	1800	590	571	571
query18	2111	283	295	283
query19	372	156	145	145
query20	118	110	142	110
query21	207	103	100	100
query22	4655	4373	4414	4373
query23	35306	34268	34365	34268
query24	11516	2505	2609	2505
query25	673	383	373	373
query26	1833	151	151	151
query27	2855	278	281	278
query28	8058	2425	2422	2422
query29	1025	422	417	417
query30	314	149	153	149
query31	1043	816	814	814
query32	98	54	55	54
query33	780	296	283	283
query34	1019	503	531	503
query35	851	710	717	710
query36	1081	958	979	958
query37	271	76	73	73
query38	4346	4296	4209	4209
query39	1471	1431	1460	1431
query40	281	107	101	101
query41	46	44	46	44
query42	109	98	98	98
query43	537	484	491	484
query44	1301	813	813	813
query45	187	164	165	164
query46	1154	704	713	704
query47	1982	1836	1822	1822
query48	411	309	324	309
query49	1279	394	389	389
query50	816	406	384	384
query51	7361	7115	7071	7071
query52	97	89	87	87
query53	251	180	182	180
query54	1261	399	420	399
query55	98	76	80	76
query56	257	257	234	234
query57	1301	1197	1193	1193
query58	235	214	216	214
query59	3316	3097	3156	3097
query60	274	254	241	241
query61	110	105	110	105
query62	873	671	695	671
query63	213	188	186	186
query64	5053	651	635	635
query65	3337	3254	3262	3254
query66	1424	321	312	312
query67	16213	15928	15548	15548
query68	4485	564	559	559
query69	401	255	310	255
query70	1180	1155	1103	1103
query71	337	262	251	251
query72	6435	4038	4021	4021
query73	756	348	356	348
query74	10477	9026	9108	9026
query75	3527	2694	2695	2694
query76	2834	1101	1092	1092
query77	507	266	291	266
query78	10623	9511	9493	9493
query79	1629	583	598	583
query80	1226	442	432	432
query81	551	230	243	230
query82	981	124	117	117
query83	253	151	148	148
query84	235	73	78	73
query85	1348	303	301	301
query86	367	300	303	300
query87	4712	4695	4658	4658
query88	3543	2223	2191	2191
query89	408	301	289	289
query90	2117	190	185	185
query91	135	108	104	104
query92	61	50	51	50
query93	1293	539	556	539
query94	1133	295	301	295
query95	353	245	249	245
query96	624	284	278	278
query97	2901	2742	2718	2718
query98	216	205	197	197
query99	1553	1339	1295	1295
Total cold run time: 304963 ms
Total hot run time: 192019 ms

doris-robot · 2024-11-20T13:03:06Z

TeamCity be ut coverage result:
Function Coverage: 38.03% (9902/26039)
Line Coverage: 29.22% (82841/283546)
Region Coverage: 28.35% (42550/150085)
Branch Coverage: 24.91% (21566/86592)
Coverage Report: http://coverage.selectdb-in.cc/coverage/085de8b2f3b2136fc6a4b9745c25da01db2ef5cb_085de8b2f3b2136fc6a4b9745c25da01db2ef5cb/report/index.html

doris-robot · 2024-11-20T13:06:34Z

ClickBench: Total hot run time: 32.99 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 085de8b2f3b2136fc6a4b9745c25da01db2ef5cb, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.06
query3	0.23	0.09	0.07
query4	1.62	0.11	0.11
query5	0.43	0.42	0.42
query6	1.16	0.68	0.66
query7	0.01	0.02	0.01
query8	0.04	0.03	0.02
query9	0.56	0.52	0.50
query10	0.56	0.57	0.56
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.61	0.58
query14	2.71	2.72	2.75
query15	0.91	0.83	0.83
query16	0.40	0.39	0.38
query17	1.07	1.08	1.08
query18	0.22	0.22	0.22
query19	1.91	1.86	2.06
query20	0.01	0.01	0.01
query21	15.37	0.60	0.58
query22	2.66	2.01	2.09
query23	16.87	0.94	0.89
query24	2.78	0.91	1.77
query25	0.34	0.13	0.08
query26	0.42	0.14	0.15
query27	0.03	0.04	0.04
query28	10.59	1.10	1.07
query29	12.58	3.21	3.23
query30	0.25	0.06	0.06
query31	2.86	0.39	0.37
query32	3.28	0.47	0.47
query33	2.98	3.10	3.02
query34	17.10	4.50	4.56
query35	4.55	4.55	4.55
query36	0.64	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.03
query40	0.16	0.12	0.12
query41	0.07	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.64 s
Total hot run time: 32.99 s

zhannngchen · 2024-11-21T06:30:47Z

please update "Problem Summary"

Yukang-Lian · 2024-11-21T12:15:44Z

run buildall

github-actions · 2024-11-21T12:19:49Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-11-21T12:40:07Z

TeamCity be ut coverage result:
Function Coverage: 38.04% (9902/26033)
Line Coverage: 29.22% (82847/283517)
Region Coverage: 28.35% (42546/150089)
Branch Coverage: 24.91% (21570/86594)
Coverage Report: http://coverage.selectdb-in.cc/coverage/02ba679f13d66c6b0b5374eb9e0a379c254d9cc6_02ba679f13d66c6b0b5374eb9e0a379c254d9cc6/report/index.html

doris-robot · 2024-11-21T12:52:08Z

TPC-H: Total hot run time: 39731 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 02ba679f13d66c6b0b5374eb9e0a379c254d9cc6, data reload: false

------ Round 1 ----------------------------------
q1	17579	7477	7294	7294
q2	2065	182	173	173
q3	10558	1111	1155	1111
q4	10561	742	791	742
q5	7589	2684	2681	2681
q6	239	147	147	147
q7	971	620	619	619
q8	9232	1835	1941	1835
q9	6532	6466	6327	6327
q10	6997	2355	2287	2287
q11	479	261	254	254
q12	422	220	213	213
q13	17781	3013	3013	3013
q14	247	214	221	214
q15	581	540	511	511
q16	669	577	606	577
q17	973	561	564	561
q18	7307	6661	6674	6661
q19	1340	1026	915	915
q20	480	184	178	178
q21	4328	3112	3167	3112
q22	380	312	306	306
Total cold run time: 107310 ms
Total hot run time: 39731 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7275	7244	7276	7244
q2	331	230	229	229
q3	2876	2743	2919	2743
q4	2125	1868	1900	1868
q5	5717	5696	5619	5619
q6	231	143	149	143
q7	2237	1830	1821	1821
q8	3518	3568	3512	3512
q9	8797	8963	8960	8960
q10	3609	3543	3571	3543
q11	593	506	511	506
q12	846	644	677	644
q13	12153	3315	3251	3251
q14	311	274	265	265
q15	582	537	534	534
q16	692	637	633	633
q17	1859	1665	1631	1631
q18	8396	7857	7598	7598
q19	1710	1633	1569	1569
q20	2109	1875	1919	1875
q21	5546	5443	5470	5443
q22	649	530	547	530
Total cold run time: 72162 ms
Total hot run time: 60161 ms

doris-robot · 2024-11-21T13:04:48Z

TPC-DS: Total hot run time: 196709 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 02ba679f13d66c6b0b5374eb9e0a379c254d9cc6, data reload: false

query1	1239	958	906	906
query2	6218	2047	2095	2047
query3	10945	4046	4124	4046
query4	67284	27918	23612	23612
query5	4935	461	430	430
query6	397	181	173	173
query7	5517	309	292	292
query8	318	227	225	225
query9	8450	2669	2663	2663
query10	453	253	256	253
query11	17128	15196	15908	15196
query12	159	101	101	101
query13	1489	409	411	409
query14	10986	7203	6777	6777
query15	228	197	182	182
query16	7083	476	501	476
query17	1101	593	570	570
query18	1890	293	299	293
query19	196	149	149	149
query20	120	109	113	109
query21	198	98	104	98
query22	4901	4490	4622	4490
query23	34729	34231	34812	34231
query24	5458	2577	2624	2577
query25	494	393	407	393
query26	642	150	144	144
query27	1750	282	325	282
query28	4460	2475	2463	2463
query29	682	430	427	427
query30	223	160	153	153
query31	1054	829	830	829
query32	63	52	53	52
query33	397	277	316	277
query34	937	524	513	513
query35	893	749	736	736
query36	1125	997	953	953
query37	122	74	84	74
query38	4525	4424	4432	4424
query39	1499	1504	1496	1496
query40	215	98	109	98
query41	46	43	42	42
query42	108	103	100	100
query43	545	494	495	494
query44	1221	835	873	835
query45	187	170	173	170
query46	1165	713	703	703
query47	2097	1970	1982	1970
query48	420	320	316	316
query49	720	381	403	381
query50	846	412	403	403
query51	7502	7242	7141	7141
query52	99	88	86	86
query53	258	177	176	176
query54	505	391	393	391
query55	82	74	86	74
query56	257	230	236	230
query57	1339	1212	1159	1159
query58	234	222	226	222
query59	3248	3001	3213	3001
query60	290	264	260	260
query61	135	129	126	126
query62	797	671	683	671
query63	218	191	201	191
query64	1501	778	633	633
query65	3269	3223	3207	3207
query66	702	286	304	286
query67	16042	15793	15864	15793
query68	3677	584	587	584
query69	411	248	264	248
query70	1144	1153	1143	1143
query71	335	248	253	248
query72	6386	3991	4094	3991
query73	777	359	359	359
query74	10059	8968	9095	8968
query75	3496	2669	2722	2669
query76	1761	1140	1128	1128
query77	499	278	268	268
query78	10614	9529	9343	9343
query79	1861	587	593	587
query80	1362	422	426	422
query81	521	243	232	232
query82	1331	117	118	117
query83	275	155	145	145
query84	275	77	70	70
query85	1016	295	296	295
query86	413	304	313	304
query87	4972	4645	4519	4519
query88	3682	2260	2216	2216
query89	418	293	315	293
query90	1895	185	184	184
query91	139	102	100	100
query92	67	51	48	48
query93	2497	546	541	541
query94	761	308	300	300
query95	345	244	241	241
query96	635	281	276	276
query97	2899	2686	2723	2686
query98	219	207	200	200
query99	2004	1304	1307	1304
Total cold run time: 321082 ms
Total hot run time: 196709 ms

doris-robot · 2024-11-21T13:10:17Z

ClickBench: Total hot run time: 32.85 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 02ba679f13d66c6b0b5374eb9e0a379c254d9cc6, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.04	0.03
query3	0.23	0.07	0.08
query4	1.62	0.10	0.11
query5	0.43	0.42	0.43
query6	1.17	0.65	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.51	0.51
query10	0.54	0.56	0.56
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.62	0.60	0.60
query14	2.82	2.82	2.69
query15	0.89	0.82	0.83
query16	0.38	0.37	0.40
query17	0.99	1.04	1.05
query18	0.22	0.23	0.20
query19	1.95	1.78	2.02
query20	0.02	0.01	0.01
query21	15.35	0.58	0.58
query22	2.87	2.13	1.97
query23	16.89	0.90	0.82
query24	2.79	1.17	1.64
query25	0.26	0.21	0.06
query26	0.53	0.14	0.14
query27	0.04	0.04	0.05
query28	10.19	1.10	1.07
query29	12.57	3.27	3.26
query30	0.26	0.06	0.06
query31	2.84	0.38	0.36
query32	3.31	0.47	0.47
query33	3.09	2.98	3.04
query34	17.07	4.46	4.49
query35	4.51	4.63	4.50
query36	0.65	0.48	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.61 s
Total hot run time: 32.85 s

github-actions · 2024-11-21T15:19:03Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-21T15:19:05Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

…paction (#44359) Problem: For a duplicate table with the following distribution, if it has already completed cumulative compaction and then undergoes full compaction, it will cause a BE core issue. Check failed: new_point == Tablet::K_INVALID_CUMULATIVE_POINT || new_point >= _cumulative_point Unexpected cumulative point: 1087, origin: 2801. "rowsets": [ "[0-386] 0 DATA NONOVERLAPPING 02000000000198aabe4290f2b0f5f35610c08a233a061892 0", "[387-387] 0 DELETE OVERLAP_UNKNOWN 0200000000541310ac4d76e7580a708a2823a4d7a4f06090 0", "[388-388] 0 DELETE OVERLAP_UNKNOWN 0200000000541d76ac4d76e7580a708a2823a4d7a4f06090 0", "[389-389] 0 DELETE OVERLAP_UNKNOWN 0200000000543b4dac4d76e7580a708a2823a4d7a4f06090 0", "[390-390] 0 DELETE OVERLAP_UNKNOWN 02000000005453aeac4d76e7580a708a2823a4d7a4f06090 0", "[391-391] 0 DELETE OVERLAP_UNKNOWN 0200000000546a44ac4d76e7580a708a2823a4d7a4f06090 0", "[392-392] 0 DELETE OVERLAP_UNKNOWN 02000000005480dbac4d76e7580a708a2823a4d7a4f06090 0", "[393-393] 0 DELETE OVERLAP_UNKNOWN 0200000000548cb3ac4d76e7580a708a2823a4d7a4f06090 0", "[394-394] 0 DELETE OVERLAP_UNKNOWN 0200000000549a25ac4d76e7580a708a2823a4d7a4f06090 0", "[395-395] 0 DELETE OVERLAP_UNKNOWN 020000000054b359ac4d76e7580a708a2823a4d7a4f06090 0", "[396-396] 0 DELETE OVERLAP_UNKNOWN 020000000054c19dac4d76e7580a708a2823a4d7a4f06090 0", "[397-397] 0 DELETE OVERLAP_UNKNOWN 020000000054d757ac4d76e7580a708a2823a4d7a4f06090 0", ... "[1085-1085] 0 DELETE OVERLAP_UNKNOWN 02000000002a0b20bd4798638f237008ff42fbca276b52a2 0", "[1087-1506] 1 DATA NONOVERLAPPING 020000000000047e3b452de14ceaad2e78a87526026d2290 326.10 KB", "[1087-1506] 1 DATA NONOVERLAPPING 020000000000047e3b452de14ceaad2e78a87526026d2290 326.10 KB", ... "[2800-2800] 0 DELETE OVERLAP_UNKNOWN 02000000002f12d6bd4798638f237008ff42fbca276b52a2 0" Reason: The duplicate table will go through ordered data compaction. Due to the special distribution of the table, the input rowset will be cut by the ordered data compaction, resulting in the full compaction only being performed on a part of the rowsets. Solution: For full compaction, prohibit ordered data compaction.

…paction (apache#44359) Problem: For a duplicate table with the following distribution, if it has already completed cumulative compaction and then undergoes full compaction, it will cause a BE core issue. Check failed: new_point == Tablet::K_INVALID_CUMULATIVE_POINT || new_point >= _cumulative_point Unexpected cumulative point: 1087, origin: 2801. "rowsets": [ "[0-386] 0 DATA NONOVERLAPPING 02000000000198aabe4290f2b0f5f35610c08a233a061892 0", "[387-387] 0 DELETE OVERLAP_UNKNOWN 0200000000541310ac4d76e7580a708a2823a4d7a4f06090 0", "[388-388] 0 DELETE OVERLAP_UNKNOWN 0200000000541d76ac4d76e7580a708a2823a4d7a4f06090 0", "[389-389] 0 DELETE OVERLAP_UNKNOWN 0200000000543b4dac4d76e7580a708a2823a4d7a4f06090 0", "[390-390] 0 DELETE OVERLAP_UNKNOWN 02000000005453aeac4d76e7580a708a2823a4d7a4f06090 0", "[391-391] 0 DELETE OVERLAP_UNKNOWN 0200000000546a44ac4d76e7580a708a2823a4d7a4f06090 0", "[392-392] 0 DELETE OVERLAP_UNKNOWN 02000000005480dbac4d76e7580a708a2823a4d7a4f06090 0", "[393-393] 0 DELETE OVERLAP_UNKNOWN 0200000000548cb3ac4d76e7580a708a2823a4d7a4f06090 0", "[394-394] 0 DELETE OVERLAP_UNKNOWN 0200000000549a25ac4d76e7580a708a2823a4d7a4f06090 0", "[395-395] 0 DELETE OVERLAP_UNKNOWN 020000000054b359ac4d76e7580a708a2823a4d7a4f06090 0", "[396-396] 0 DELETE OVERLAP_UNKNOWN 020000000054c19dac4d76e7580a708a2823a4d7a4f06090 0", "[397-397] 0 DELETE OVERLAP_UNKNOWN 020000000054d757ac4d76e7580a708a2823a4d7a4f06090 0", ... "[1085-1085] 0 DELETE OVERLAP_UNKNOWN 02000000002a0b20bd4798638f237008ff42fbca276b52a2 0", "[1087-1506] 1 DATA NONOVERLAPPING 020000000000047e3b452de14ceaad2e78a87526026d2290 326.10 KB", "[1087-1506] 1 DATA NONOVERLAPPING 020000000000047e3b452de14ceaad2e78a87526026d2290 326.10 KB", ... "[2800-2800] 0 DELETE OVERLAP_UNKNOWN 02000000002f12d6bd4798638f237008ff42fbca276b52a2 0" Reason: The duplicate table will go through ordered data compaction. Due to the special distribution of the table, the input rowset will be cut by the ordered data compaction, resulting in the full compaction only being performed on a part of the rowsets. Solution: For full compaction, prohibit ordered data compaction.

…n should not do ordered data compaction (#44359)" (#44529)

…red data compaction #44359 (#44432) Cherry-picked from #44359 Co-authored-by: abmdocrt <lianyukang@selectdb.com>

Yukang-Lian added 3 commits November 21, 2024 15:36

1

d5e67ef

2

7166bd5

3

02ba679

Yukang-Lian force-pushed the Fix-Full-Compaction-1 branch from 085de8b to 02ba679 Compare November 21, 2024 12:14

gavinchou approved these changes Nov 21, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 21, 2024

github-actions bot added the reviewed label Nov 21, 2024

luwei16 approved these changes Nov 21, 2024

View reviewed changes

dataroaring added dev/3.0.x dev/2.1.x labels Nov 21, 2024

dataroaring approved these changes Nov 21, 2024

View reviewed changes

dataroaring merged commit ca579c1 into apache:master Nov 21, 2024
31 of 34 checks passed

github-actions bot mentioned this pull request Nov 21, 2024

branch-3.0: [Fix](full compaction) Full compaction should not do ordered data compaction #44359 #44432

Merged

github-actions bot mentioned this pull request Nov 21, 2024

branch-2.1: [Fix](full compaction) Full compaction should not do ordered data compaction #44359 #44433

Closed

Yukang-Lian mentioned this pull request Nov 25, 2024

[cherry-pick](branch-2.1) Pick "[Fix](full compaction) Full compaction should not do ordered data compaction (#44359)" #44529

Merged

16 tasks

yiguolei pushed a commit that referenced this pull request Nov 26, 2024

[cherry-pick](branch-2.1) Pick "[Fix](full compaction) Full compactio…

550aa9c

…n should not do ordered data compaction (#44359)" (#44529)

yiguolei added dev/2.1.8-merged and removed dev/2.1.x labels Nov 26, 2024

yiguolei pushed a commit that referenced this pull request Nov 26, 2024

branch-3.0: [Fix](full compaction) Full compaction should not do orde…

98e8b2c

…red data compaction #44359 (#44432) Cherry-picked from #44359 Co-authored-by: abmdocrt <lianyukang@selectdb.com>

yiguolei added dev/3.0.4-merged and removed dev/3.0.x labels Nov 26, 2024

gavinchou mentioned this pull request Feb 18, 2025

Release Notes 3.0.4 #48013

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fix](full compaction) Full compaction should not do ordered data compaction #44359

[Fix](full compaction) Full compaction should not do ordered data compaction #44359

Yukang-Lian commented Nov 20, 2024 •

edited

Loading

doris-robot commented Nov 20, 2024

Yukang-Lian commented Nov 20, 2024

Yukang-Lian commented Nov 20, 2024

github-actions bot commented Nov 20, 2024

doris-robot commented Nov 20, 2024

doris-robot commented Nov 20, 2024

doris-robot commented Nov 20, 2024

doris-robot commented Nov 20, 2024

zhannngchen commented Nov 21, 2024

Yukang-Lian commented Nov 21, 2024

github-actions bot commented Nov 21, 2024

doris-robot commented Nov 21, 2024

doris-robot commented Nov 21, 2024

doris-robot commented Nov 21, 2024

doris-robot commented Nov 21, 2024

github-actions bot commented Nov 21, 2024

github-actions bot commented Nov 21, 2024

dataroaring left a comment

[Fix](full compaction) Full compaction should not do ordered data compaction #44359

[Fix](full compaction) Full compaction should not do ordered data compaction #44359

Conversation

Yukang-Lian commented Nov 20, 2024 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented Nov 20, 2024

Yukang-Lian commented Nov 20, 2024

Yukang-Lian commented Nov 20, 2024

github-actions bot commented Nov 20, 2024

doris-robot commented Nov 20, 2024

doris-robot commented Nov 20, 2024

doris-robot commented Nov 20, 2024

doris-robot commented Nov 20, 2024

zhannngchen commented Nov 21, 2024

Yukang-Lian commented Nov 21, 2024

github-actions bot commented Nov 21, 2024

doris-robot commented Nov 21, 2024

doris-robot commented Nov 21, 2024

doris-robot commented Nov 21, 2024

doris-robot commented Nov 21, 2024

github-actions bot commented Nov 21, 2024

github-actions bot commented Nov 21, 2024

dataroaring left a comment

Choose a reason for hiding this comment

Yukang-Lian commented Nov 20, 2024 •

edited

Loading