Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[cherry-pick](branch-2.0) fix data correctness risk when load delete sign data into a table with sequence col (#32574) #36228

Merged
merged 1 commit into from
Jun 13, 2024
Merged

[cherry-pick](branch-2.0) fix data correctness risk when load delete sign data into a table with sequence col (#32574) #36228

merged 1 commit into from
Jun 13, 2024

Conversation

zhannngchen
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

cherry-pick #32574

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhannngchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.80% (8094/21411)
Line Coverage: 29.47% (66125/224384)
Region Coverage: 28.94% (34084/117781)
Branch Coverage: 24.80% (17493/70524)
Coverage Report: http://coverage.selectdb-in.cc/coverage/744ea52db84562a23eee6698a49766e986e279d8_744ea52db84562a23eee6698a49766e986e279d8/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 49780 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 744ea52db84562a23eee6698a49766e986e279d8, data reload: false

------ Round 1 ----------------------------------
q1	17554	4410	4320	4320
q2	2070	160	151	151
q3	10471	1905	1929	1905
q4	10331	1257	1343	1257
q5	8456	3854	3884	3854
q6	237	125	125	125
q7	2052	1605	1624	1605
q8	9532	2724	2703	2703
q9	13870	11099	10184	10184
q10	8695	3516	3513	3513
q11	408	247	256	247
q12	468	302	306	302
q13	18340	3963	4040	3963
q14	363	331	322	322
q15	503	460	472	460
q16	692	581	594	581
q17	1127	926	929	926
q18	7183	6864	6993	6864
q19	1747	1656	1644	1644
q20	539	322	304	304
q21	4393	4103	4116	4103
q22	529	461	447	447
Total cold run time: 119560 ms
Total hot run time: 49780 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4328	4296	4394	4296
q2	324	235	226	226
q3	4168	4094	4173	4094
q4	2759	2744	2734	2734
q5	7139	7100	7090	7090
q6	239	120	121	120
q7	3255	2805	2833	2805
q8	4311	4454	4497	4454
q9	16916	16784	16781	16781
q10	4204	4229	4263	4229
q11	756	705	699	699
q12	1014	872	860	860
q13	6800	3776	3718	3718
q14	456	421	428	421
q15	500	448	454	448
q16	728	683	684	683
q17	3848	3787	3872	3787
q18	8825	8900	8836	8836
q19	1740	1682	1711	1682
q20	2396	2169	2120	2120
q21	8496	8465	8572	8465
q22	1074	982	1003	982
Total cold run time: 84276 ms
Total hot run time: 79530 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 204393 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 744ea52db84562a23eee6698a49766e986e279d8, data reload: false

query1	968	389	385	385
query2	6549	2922	2717	2717
query3	6926	203	197	197
query4	20908	18141	18065	18065
query5	19741	6529	6527	6527
query6	286	224	225	224
query7	4162	306	317	306
query8	419	418	467	418
query9	3106	2666	2608	2608
query10	417	300	285	285
query11	11350	10666	10662	10662
query12	123	78	76	76
query13	5632	685	698	685
query14	17823	13389	13570	13389
query15	356	252	252	252
query16	6485	293	267	267
query17	1690	1446	908	908
query18	2295	408	421	408
query19	211	152	158	152
query20	84	82	80	80
query21	195	95	95	95
query22	5297	5200	5157	5157
query23	32468	31802	31850	31802
query24	6977	6487	6480	6480
query25	535	443	424	424
query26	533	162	162	162
query27	1885	289	293	289
query28	6224	2366	2337	2337
query29	2861	2770	2685	2685
query30	249	168	173	168
query31	894	757	768	757
query32	75	64	62	62
query33	401	256	270	256
query34	862	483	468	468
query35	1118	857	861	857
query36	1193	1304	1455	1304
query37	91	62	60	60
query38	3104	2946	2942	2942
query39	1380	1342	1321	1321
query40	213	101	95	95
query41	47	45	44	44
query42	89	84	86	84
query43	703	649	691	649
query44	1174	717	714	714
query45	249	239	239	239
query46	1225	975	986	975
query47	1843	1795	2029	1795
query48	1006	709	692	692
query49	632	377	376	376
query50	861	632	629	629
query51	4767	4700	4731	4700
query52	94	84	78	78
query53	454	321	333	321
query54	2668	2468	2495	2468
query55	81	85	75	75
query56	251	215	212	212
query57	1266	1135	1084	1084
query58	233	216	186	186
query59	4155	4318	3670	3670
query60	218	195	199	195
query61	100	95	99	95
query62	857	491	510	491
query63	488	338	336	336
query64	2326	1547	1510	1510
query65	3666	3550	3579	3550
query66	842	379	375	375
query67	17327	16819	15590	15590
query68	5912	665	653	653
query69	559	337	352	337
query70	1394	1474	1724	1474
query71	385	318	315	315
query72	6389	3494	3506	3494
query73	731	322	312	312
query74	6325	5887	5931	5887
query75	4513	3695	3639	3639
query76	3543	1164	1155	1155
query77	553	252	253	252
query78	12523	11834	11657	11657
query79	7550	631	629	629
query80	2658	409	405	405
query81	546	242	236	236
query82	1582	100	96	96
query83	239	134	132	132
query84	268	74	71	71
query85	1366	328	325	325
query86	351	294	288	288
query87	3229	2979	3042	2979
query88	5014	2348	2332	2332
query89	342	294	330	294
query90	1750	211	213	211
query91	169	145	146	145
query92	56	54	54	54
query93	3139	592	571	571
query94	907	208	197	197
query95	1091	1056	1047	1047
query96	639	324	328	324
query97	6537	6361	6445	6361
query98	187	171	167	167
query99	2963	900	889	889
Total cold run time: 307118 ms
Total hot run time: 204393 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.89 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 744ea52db84562a23eee6698a49766e986e279d8, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.04	0.04
query4	1.78	0.05	0.06
query5	0.53	0.53	0.52
query6	1.29	0.62	0.63
query7	0.02	0.01	0.01
query8	0.03	0.02	0.03
query9	0.50	0.48	0.47
query10	0.56	0.53	0.51
query11	0.12	0.08	0.09
query12	0.12	0.09	0.09
query13	0.62	0.62	0.60
query14	0.81	0.78	0.77
query15	0.79	0.75	0.76
query16	0.38	0.37	0.39
query17	1.03	1.03	1.02
query18	0.24	0.24	0.24
query19	1.87	1.85	1.86
query20	0.01	0.01	0.01
query21	15.49	0.58	0.57
query22	2.04	2.11	1.59
query23	17.32	0.99	0.88
query24	6.33	1.60	1.32
query25	0.37	0.11	0.06
query26	0.78	0.16	0.15
query27	0.04	0.03	0.04
query28	5.90	0.77	0.77
query29	12.61	2.16	2.36
query30	0.55	0.53	0.51
query31	2.81	0.39	0.37
query32	3.38	0.51	0.50
query33	3.08	3.06	3.08
query34	15.25	4.79	4.79
query35	4.87	4.83	4.83
query36	1.05	1.01	1.02
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.15	0.13
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 103.29 s
Total hot run time: 30.89 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 744ea52db84562a23eee6698a49766e986e279d8 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.8 seconds inserted 10000000 Rows, about 458K ops/s

@zhannngchen zhannngchen merged commit c7acf31 into apache:branch-2.0 Jun 13, 2024
22 of 25 checks passed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@zhannngchen
[cherry-pick](branch-2.0) fix data correctness risk when load delete …
439805c 
…sign data into a table with sequence col (apache#32574) (apache#36228)

cherry-pick apache#32574
bobhan1 pushed a commit to bobhan1/doris that referenced this pull request Jan 20, 2025
@zhannngchen
[cherry-pick](branch-2.0) fix data correctness risk when load delete …
a9037dc 
…sign data into a table with sequence col (apache#32574) (apache#36228) (apache#278)

cherry-pick apache#32574
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
kind/test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@zhannngchen @doris-robot