[opt](inverted index) performance optimization for need_read_data in compound #35346

[xiaokang]

Proposed changes

pick from master #35346

Issue Number: close #xxx

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[xiaokang]

run buildall

[doris-robot]

TPC-H: Total hot run time: 49752 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4ebbddf45aa730797dfb56f0ce2d94f64d442eaf, data reload: false

------ Round 1 ----------------------------------
q1	17668	4358	4379	4358
q2	2081	153	143	143
q3	10451	1890	1946	1890
q4	10329	1219	1296	1219
q5	8480	3859	3909	3859
q6	232	127	128	127
q7	2036	1597	1625	1597
q8	9282	2704	2687	2687
q9	10541	10321	10270	10270
q10	8655	3515	3530	3515
q11	419	255	249	249
q12	465	307	302	302
q13	18347	3934	4055	3934
q14	353	324	332	324
q15	501	460	463	460
q16	670	589	573	573
q17	1121	969	970	969
q18	7316	6935	6951	6935
q19	1712	1584	1575	1575
q20	547	308	300	300
q21	4464	4094	4074	4074
q22	497	397	392	392
Total cold run time: 116167 ms
Total hot run time: 49752 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4305	4310	4311	4310
q2	323	230	222	222
q3	4160	4112	4104	4104
q4	2740	2732	2733	2732
q5	7180	7061	7076	7061
q6	239	117	120	117
q7	3231	2825	2868	2825
q8	4321	4434	4421	4421
q9	16879	16800	16641	16641
q10	4246	4261	4247	4247
q11	745	684	695	684
q12	1036	877	863	863
q13	6761	3736	3722	3722
q14	469	420	421	420
q15	501	447	449	447
q16	720	670	686	670
q17	3780	3978	3863	3863
q18	8757	8673	8695	8673
q19	1696	1698	1634	1634
q20	2381	2156	2108	2108
q21	8486	8376	8492	8376
q22	1032	974	959	959
Total cold run time: 83988 ms
Total hot run time: 79099 ms

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 37.81% (8099/21421)
Line Coverage: 29.47% (66147/224444)
Region Coverage: 28.96% (34095/117738)
Branch Coverage: 24.81% (17498/70524)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4ebbddf45aa730797dfb56f0ce2d94f64d442eaf_4ebbddf45aa730797dfb56f0ce2d94f64d442eaf/report/index.html

[doris-robot]

TPC-DS: Total hot run time: 203039 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4ebbddf45aa730797dfb56f0ce2d94f64d442eaf, data reload: false

query1	932	382	374	374
query2	6599	2860	2603	2603
query3	6919	213	204	204
query4	20107	17923	17867	17867
query5	19722	6510	6503	6503
query6	349	215	232	215
query7	4613	299	303	299
query8	270	248	259	248
query9	3136	2681	2634	2634
query10	418	292	304	292
query11	11253	10643	10572	10572
query12	119	81	74	74
query13	5591	683	707	683
query14	18485	13527	13589	13527
query15	384	227	228	227
query16	6439	294	260	260
query17	1354	1563	887	887
query18	2252	407	414	407
query19	216	151	156	151
query20	80	80	80	80
query21	190	96	96	96
query22	5042	5087	4910	4910
query23	32664	31568	32254	31568
query24	6876	6571	6509	6509
query25	521	440	420	420
query26	510	168	160	160
query27	1780	292	297	292
query28	6132	2396	2347	2347
query29	2905	2816	2765	2765
query30	248	165	169	165
query31	916	759	741	741
query32	69	62	61	61
query33	404	264	244	244
query34	864	465	475	465
query35	1129	913	931	913
query36	1215	1066	1183	1066
query37	92	62	68	62
query38	3120	2929	2931	2929
query39	1372	1310	1312	1310
query40	209	99	95	95
query41	41	37	37	37
query42	91	80	83	80
query43	764	656	628	628
query44	1123	719	717	717
query45	244	229	230	229
query46	1243	967	977	967
query47	1950	1640	1787	1640
query48	1014	691	683	683
query49	626	372	363	363
query50	868	590	624	590
query51	4766	4687	4698	4687
query52	89	91	74	74
query53	448	311	318	311
query54	2661	2428	2445	2428
query55	89	82	83	82
query56	246	222	221	221
query57	1142	1097	1108	1097
query58	205	207	199	199
query59	4197	4115	3859	3859
query60	216	209	203	203
query61	97	97	105	97
query62	833	479	458	458
query63	479	328	338	328
query64	2583	1547	1508	1508
query65	3615	3547	3556	3547
query66	764	380	363	363
query67	15949	17075	15956	15956
query68	8487	674	645	645
query69	568	337	344	337
query70	1731	1741	1363	1363
query71	401	309	310	309
query72	6529	3496	3506	3496
query73	729	329	327	327
query74	6318	5721	5880	5721
query75	4664	3620	3678	3620
query76	4741	1178	1188	1178
query77	639	257	260	257
query78	12575	12037	11449	11449
query79	8744	668	650	650
query80	2245	413	402	402
query81	507	238	233	233
query82	1452	99	95	95
query83	170	137	133	133
query84	256	70	73	70
query85	1312	324	318	318
query86	365	294	290	290
query87	3184	2969	2988	2969
query88	5019	2345	2332	2332
query89	379	336	326	326
query90	1805	206	209	206
query91	175	140	140	140
query92	57	57	55	55
query93	5291	617	593	593
query94	893	207	218	207
query95	1105	1087	1057	1057
query96	640	341	329	329
query97	6444	6445	6299	6299
query98	190	172	172	172
query99	3002	884	985	884
Total cold run time: 312456 ms
Total hot run time: 203039 ms

[doris-robot]

ClickBench: Total hot run time: 31.33 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4ebbddf45aa730797dfb56f0ce2d94f64d442eaf, data reload: false

query1	0.02	0.03	0.02
query2	0.06	0.03	0.02
query3	0.25	0.04	0.05
query4	1.79	0.07	0.06
query5	0.54	0.53	0.52
query6	1.23	0.61	0.61
query7	0.02	0.01	0.00
query8	0.03	0.02	0.02
query9	0.52	0.48	0.46
query10	0.55	0.54	0.54
query11	0.12	0.08	0.08
query12	0.12	0.09	0.09
query13	0.61	0.62	0.61
query14	0.80	0.78	0.78
query15	0.77	0.76	0.77
query16	0.38	0.39	0.39
query17	0.95	0.99	0.99
query18	0.22	0.26	0.25
query19	1.88	1.80	1.87
query20	0.02	0.00	0.01
query21	15.48	0.55	0.57
query22	2.08	2.53	1.64
query23	17.19	0.99	0.99
query24	7.05	1.51	1.91
query25	0.38	0.11	0.05
query26	0.82	0.15	0.15
query27	0.04	0.04	0.04
query28	5.26	0.71	0.70
query29	12.61	2.42	2.37
query30	0.56	0.57	0.51
query31	2.80	0.39	0.38
query32	3.39	0.49	0.48
query33	3.05	3.06	3.06
query34	15.27	4.81	4.80
query35	4.84	4.80	4.84
query36	1.07	1.01	1.01
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 103.16 s
Total hot run time: 31.33 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4ebbddf45aa730797dfb56f0ce2d94f64d442eaf with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

[xiaokang]

run buildall

[doris-robot]

TPC-H: Total hot run time: 49716 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa86a12ad0448567b5f84ad17d55062dd7b4473a, data reload: false

------ Round 1 ----------------------------------
q1	17669	4369	4350	4350
q2	2042	185	148	148
q3	10376	1880	1947	1880
q4	10365	1268	1321	1268
q5	8425	3885	3884	3884
q6	237	124	123	123
q7	2052	1605	1587	1587
q8	9309	2727	2695	2695
q9	10537	10288	10262	10262
q10	8634	3533	3492	3492
q11	436	251	250	250
q12	468	301	301	301
q13	18363	3941	4020	3941
q14	368	331	341	331
q15	527	462	460	460
q16	675	576	581	576
q17	1116	947	995	947
q18	7209	6926	6977	6926
q19	1716	1587	1560	1560
q20	535	319	293	293
q21	4378	4172	4044	4044
q22	491	402	398	398
Total cold run time: 115928 ms
Total hot run time: 49716 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4339	4300	4360	4300
q2	320	222	223	222
q3	4204	4131	4126	4126
q4	2753	2744	2744	2744
q5	7187	7082	7112	7082
q6	238	121	118	118
q7	3241	2830	2895	2830
q8	4364	4432	4459	4432
q9	16909	16729	16739	16729
q10	4253	4265	4238	4238
q11	771	689	663	663
q12	1024	860	838	838
q13	7256	3702	3744	3702
q14	452	415	432	415
q15	499	453	456	453
q16	745	678	672	672
q17	3863	3836	3839	3836
q18	8892	8834	8670	8670
q19	1722	1692	1633	1633
q20	2388	2142	2110	2110
q21	8481	8393	8467	8393
q22	1012	938	900	900
Total cold run time: 84913 ms
Total hot run time: 79106 ms

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 37.81% (8099/21421)
Line Coverage: 29.47% (66148/224444)
Region Coverage: 28.96% (34095/117738)
Branch Coverage: 24.81% (17496/70524)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fa86a12ad0448567b5f84ad17d55062dd7b4473a_fa86a12ad0448567b5f84ad17d55062dd7b4473a/report/index.html

[doris-robot]

TPC-DS: Total hot run time: 203799 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fa86a12ad0448567b5f84ad17d55062dd7b4473a, data reload: false

query1	929	389	378	378
query2	6554	2598	2481	2481
query3	6925	205	201	201
query4	20407	18114	18074	18074
query5	19728	6519	6547	6519
query6	277	215	222	215
query7	4144	303	311	303
query8	254	262	245	245
query9	3114	2679	2606	2606
query10	425	292	297	292
query11	11332	10696	10720	10696
query12	125	78	75	75
query13	5587	724	681	681
query14	17345	13355	14001	13355
query15	375	219	237	219
query16	6472	288	265	265
query17	1711	1460	874	874
query18	2320	413	405	405
query19	211	150	147	147
query20	71	82	78	78
query21	201	96	98	96
query22	5322	5143	5226	5143
query23	32494	31850	31766	31766
query24	7033	6585	6491	6491
query25	535	433	412	412
query26	628	164	157	157
query27	2045	298	294	294
query28	6121	2362	2314	2314
query29	2883	2768	2840	2768
query30	247	163	170	163
query31	905	739	735	735
query32	70	62	60	60
query33	406	255	241	241
query34	842	478	477	477
query35	1132	951	932	932
query36	1213	1303	1329	1303
query37	92	62	61	61
query38	3062	2923	2911	2911
query39	1390	1316	1339	1316
query40	220	91	97	91
query41	40	38	37	37
query42	84	81	84	81
query43	788	628	627	627
query44	1125	721	714	714
query45	242	232	230	230
query46	1230	960	956	956
query47	2062	1758	1701	1701
query48	1010	726	718	718
query49	625	381	363	363
query50	856	576	595	576
query51	4849	4623	4605	4605
query52	100	84	86	84
query53	444	320	327	320
query54	2675	2449	2444	2444
query55	79	80	78	78
query56	229	208	214	208
query57	1354	1163	1100	1100
query58	217	205	190	190
query59	4228	4066	3963	3963
query60	207	217	215	215
query61	94	93	94	93
query62	829	421	451	421
query63	487	342	336	336
query64	2498	1516	1482	1482
query65	3639	4081	3539	3539
query66	808	386	371	371
query67	17655	16210	15347	15347
query68	8552	656	689	656
query69	579	360	350	350
query70	1551	1339	1324	1324
query71	405	298	334	298
query72	6484	3467	3649	3467
query73	747	322	329	322
query74	6353	5862	5875	5862
query75	4772	3719	3694	3694
query76	4733	1170	1210	1170
query77	649	252	275	252
query78	12518	12555	12081	12081
query79	11881	671	619	619
query80	872	401	397	397
query81	504	235	239	235
query82	643	99	102	99
query83	173	134	134	134
query84	258	70	72	70
query85	867	321	328	321
query86	339	292	289	289
query87	3272	3044	3032	3032
query88	4926	2342	2348	2342
query89	363	282	304	282
query90	1892	206	211	206
query91	172	139	141	139
query92	58	54	54	54
query93	4243	582	557	557
query94	746	211	203	203
query95	1117	1062	1062	1062
query96	639	330	332	330
query97	6427	6343	6343	6343
query98	193	180	173	173
query99	2895	864	797	797
Total cold run time: 313424 ms
Total hot run time: 203799 ms

[doris-robot]

ClickBench: Total hot run time: 30.58 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fa86a12ad0448567b5f84ad17d55062dd7b4473a, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.05
query4	1.78	0.08	0.08
query5	0.54	0.52	0.52
query6	1.24	0.62	0.63
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.49	0.48
query10	0.53	0.54	0.53
query11	0.13	0.09	0.09
query12	0.11	0.09	0.09
query13	0.62	0.62	0.62
query14	0.79	0.79	0.78
query15	0.78	0.77	0.76
query16	0.36	0.36	0.36
query17	1.01	1.01	0.99
query18	0.24	0.23	0.26
query19	1.88	1.88	1.82
query20	0.02	0.01	0.01
query21	15.84	0.57	0.57
query22	2.38	2.57	1.61
query23	16.86	1.02	1.07
query24	7.12	0.83	0.92
query25	0.38	0.12	0.05
query26	0.68	0.15	0.15
query27	0.05	0.04	0.03
query28	6.27	0.73	0.74
query29	12.72	2.13	2.17
query30	0.60	0.55	0.52
query31	2.82	0.38	0.37
query32	3.38	0.51	0.49
query33	3.07	3.05	3.11
query34	15.27	4.80	4.80
query35	4.85	4.87	4.87
query36	1.05	1.03	1.01
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.01
Total cold run time: 104.67 s
Total hot run time: 30.58 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit fa86a12ad0448567b5f84ad17d55062dd7b4473a with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s