Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](ES Catalog)Support parse single value for array column #40614

Merged
merged 1 commit into from
Sep 11, 2024
Merged

[fix](ES Catalog)Support parse single value for array column #40614

merged 1 commit into from
Sep 11, 2024

Conversation

qidaye
Copy link
Contributor

@qidaye qidaye commented Sep 10, 2024

Proposed changes

Follow up #39104, when the field has one value and we map it as array type in Doris, we parse the single value to a single element array to make them queryable.

close #40406

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Sep 10, 2024

run buildall

morningman
morningman approved these changes Sep 10, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 10, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.81% (9393/25518)
Line Coverage: 28.22% (77480/274527)
Region Coverage: 27.63% (40005/144811)
Branch Coverage: 24.26% (20352/83898)
Coverage Report: http://coverage.selectdb-in.cc/coverage/46bcfc4b71c0fc20c4a66582f4ecece494b29a96_46bcfc4b71c0fc20c4a66582f4ecece494b29a96/report/index.html

eldenmoon
eldenmoon approved these changes Sep 11, 2024
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

TPC-H: Total hot run time: 38510 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 46bcfc4b71c0fc20c4a66582f4ecece494b29a96, data reload: false

------ Round 1 ----------------------------------
q1	17935	4480	4416	4416
q2	2013	187	178	178
q3	10476	1115	1123	1115
q4	10136	692	720	692
q5	7714	2826	2779	2779
q6	227	143	140	140
q7	976	638	604	604
q8	9319	2052	2075	2052
q9	7283	6531	6539	6531
q10	7036	2214	2199	2199
q11	475	253	252	252
q12	401	227	230	227
q13	18952	3146	3139	3139
q14	285	242	246	242
q15	558	482	485	482
q16	521	421	425	421
q17	986	661	740	661
q18	7491	6902	6887	6887
q19	1401	1118	1130	1118
q20	700	353	328	328
q21	4565	3180	3038	3038
q22	1127	1009	1013	1009
Total cold run time: 110577 ms
Total hot run time: 38510 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4452	4306	4298	4298
q2	371	274	272	272
q3	2939	2692	2637	2637
q4	1949	1720	1690	1690
q5	5390	5378	5397	5378
q6	240	133	134	133
q7	2149	1774	1727	1727
q8	3235	3410	3388	3388
q9	8446	8481	8500	8481
q10	3457	3228	3190	3190
q11	613	492	506	492
q12	813	603	621	603
q13	11839	3129	3100	3100
q14	313	298	267	267
q15	533	474	482	474
q16	510	472	461	461
q17	1811	1512	1505	1505
q18	7747	7361	7469	7361
q19	1696	1608	1584	1584
q20	2094	1856	1867	1856
q21	5321	5220	5250	5220
q22	1100	1046	1057	1046
Total cold run time: 67018 ms
Total hot run time: 55163 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192657 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 46bcfc4b71c0fc20c4a66582f4ecece494b29a96, data reload: false

query1	910	381	391	381
query2	6470	1944	1978	1944
query3	6646	205	223	205
query4	34016	23258	23181	23181
query5	4173	525	499	499
query6	254	169	172	169
query7	4592	319	309	309
query8	288	233	224	224
query9	8614	2487	2485	2485
query10	451	284	272	272
query11	17718	15224	15464	15224
query12	157	99	95	95
query13	1624	395	364	364
query14	9644	7340	6995	6995
query15	280	172	179	172
query16	7875	432	407	407
query17	1591	581	541	541
query18	2031	285	285	285
query19	185	143	147	143
query20	119	114	114	114
query21	205	105	104	104
query22	4480	4200	4212	4200
query23	34304	33664	33780	33664
query24	11253	2969	2836	2836
query25	635	410	412	410
query26	1167	157	161	157
query27	2405	280	289	280
query28	7306	2055	2022	2022
query29	834	437	432	432
query30	309	162	157	157
query31	1005	790	827	790
query32	98	58	59	58
query33	766	304	302	302
query34	970	490	488	488
query35	858	731	718	718
query36	1100	917	939	917
query37	163	88	90	88
query38	4075	3929	3881	3881
query39	1480	1408	1437	1408
query40	200	119	120	119
query41	51	49	50	49
query42	122	98	99	98
query43	517	489	461	461
query44	1214	773	750	750
query45	200	170	175	170
query46	1103	729	782	729
query47	1877	1777	1804	1777
query48	393	306	309	306
query49	1117	465	464	464
query50	829	418	419	418
query51	7059	6949	6851	6851
query52	104	90	88	88
query53	265	187	185	185
query54	1067	469	475	469
query55	76	75	75	75
query56	300	274	284	274
query57	1194	1087	1084	1084
query58	249	249	246	246
query59	3153	2833	2899	2833
query60	301	274	278	274
query61	105	104	98	98
query62	825	652	658	652
query63	222	193	189	189
query64	5367	691	677	677
query65	3317	3163	3165	3163
query66	1423	349	334	334
query67	15917	15291	15269	15269
query68	3117	550	557	550
query69	410	272	275	272
query70	1151	1106	1014	1014
query71	339	272	269	269
query72	6223	4000	3901	3901
query73	740	325	335	325
query74	9459	8838	8883	8838
query75	3408	2735	2751	2735
query76	1892	1003	934	934
query77	461	314	336	314
query78	9817	9301	10436	9301
query79	931	878	855	855
query80	846	814	798	798
query81	499	261	261	261
query82	278	266	261	261
query83	189	196	191	191
query84	233	108	108	108
query85	725	396	386	386
query86	326	325	313	313
query87	4317	4328	4336	4328
query88	4707	4116	4111	4111
query89	368	366	369	366
query90	1874	388	316	316
query91	128	132	123	123
query92	81	75	76	75
query93	914	936	900	900
query94	840	372	375	372
query95	478	433	419	419
query96	471	470	472	470
query97	3124	3105	3069	3069
query98	240	225	220	220
query99	1398	1273	1239	1239
Total cold run time: 289602 ms
Total hot run time: 192657 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 46bcfc4b71c0fc20c4a66582f4ecece494b29a96, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.08
query5	0.51	0.49	0.49
query6	1.14	0.74	0.73
query7	0.01	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.54	0.56	0.53
query11	0.15	0.13	0.12
query12	0.16	0.13	0.13
query13	0.60	0.60	0.59
query14	1.40	1.43	1.37
query15	0.84	0.84	0.82
query16	0.36	0.37	0.37
query17	1.04	0.98	1.04
query18	0.22	0.20	0.21
query19	2.01	1.86	1.88
query20	0.01	0.01	0.00
query21	15.40	0.67	0.66
query22	4.00	6.92	2.68
query23	18.26	1.43	1.40
query24	2.08	0.23	0.22
query25	0.14	0.09	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.07
query28	13.24	1.02	0.99
query29	12.67	3.34	3.32
query30	0.24	0.06	0.05
query31	2.88	0.40	0.39
query32	3.26	0.49	0.47
query33	2.96	2.99	3.01
query34	17.04	4.39	4.43
query35	4.42	4.45	4.47
query36	0.65	0.48	0.48
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.10	0.05	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.96 s
Total hot run time: 32.37 s

@qidaye qidaye merged commit d8c34be into apache:master Sep 11, 2024
24 of 29 checks passed
@qidaye qidaye deleted the fix_array_and_string branch September 11, 2024 06:27
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Sep 11, 2024
@qidaye
[fix](ES Catalog)Support parse single value for array column (apache#…
09b7005 
…40614)

Follow up apache#39104, when the field has one value and we map it as array
type in Doris, we parse the single value to a single element array to
make them queryable.

close apache#40406
@qidaye qidaye mentioned this pull request Sep 11, 2024
Merged
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Sep 11, 2024
@qidaye
[fix](ES Catalog)Support parse single value for array column (apache#…
5e47e46 
…40614)

Follow up apache#39104, when the field has one value and we map it as array
type in Doris, we parse the single value to a single element array to
make them queryable.

close apache#40406
@qidaye qidaye mentioned this pull request Sep 11, 2024
Merged
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Sep 11, 2024
@qidaye
[fix](ES Catalog)Support parse single value for array column (apache#…
e2bb504 
…40614)

Follow up apache#39104, when the field has one value and we map it as array
type in Doris, we parse the single value to a single element array to
make them queryable.

close apache#40406
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Sep 11, 2024
@qidaye
[fix](ES Catalog)Support parse single value for array column (apache#…
7afd313 
…40614)

Follow up apache#39104, when the field has one value and we map it as array
type in Doris, we parse the single value to a single element array to
make them queryable.

close apache#40406
xiaokang pushed a commit that referenced this pull request Sep 11, 2024
@qidaye
[fix](ES Catalog)Support parse single value for array column (#40614) (…
c63359a 
…#40666)
yiguolei pushed a commit that referenced this pull request Sep 11, 2024
@qidaye
[fix](ES Catalog)Support parse single value for array column (#40614) (…
8708fae 
…#40660)

bp #40614
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@qidaye @dataroaring
[fix](ES Catalog)Support parse single value for array column (#40614)
00fa2d9 
Follow up #39104, when the field has one value and we map it as array
type in Doris, we parse the single value to a single element array to
make them queryable.

close #40406
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@eldenmoon eldenmoon eldenmoon approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.7-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] ES Catalog Query Error
7 participants
@qidaye @doris-robot @morningman @eldenmoon @xiaokang @wm1581066 @dataroaring