Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature](pipelineX) add local_shuffle in sort partition sort analytic node #28265

Merged

Conversation

Mryange
Copy link
Contributor

@Mryange Mryange commented Dec 12, 2023

Proposed changes

test in
#28254
#28248
#28146

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@Mryange
Copy link
Contributor Author

Mryange commented Dec 12, 2023

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@@ -105,6 +105,12 @@ class PartitionSortSinkOperatorX final : public DataSinkOperatorX<PartitionSortS
Status open(RuntimeState* state) override;
Status sink(RuntimeState* state, vectorized::Block* in_block,
SourceState source_state) override;
ExchangeType get_local_exchange_type() const override {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: method 'get_local_exchange_type' can be made static [readability-convert-member-functions-to-static]

Suggested change
ExchangeType get_local_exchange_type() const override {
static ExchangeType get_local_exchange_type() override {

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.08 seconds
stream load tsv: 597 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17224361124 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit c28f48966a0940ded8f9cfc060289c5319e218c5, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4648	4446	4446	4446
q2	369	146	162	146
q3	1456	1229	1184	1184
q4	1117	899	883	883
q5	3117	3180	3130	3130
q6	249	124	128	124
q7	1015	484	475	475
q8	2212	2233	2208	2208
q9	6656	6676	6679	6676
q10	3190	3256	3261	3256
q11	328	208	209	208
q12	347	201	206	201
q13	4625	3854	3834	3834
q14	243	210	210	210
q15	562	526	521	521
q16	436	388	401	388
q17	1018	555	514	514
q18	8999	7889	8763	7889
q19	1520	1269	1396	1269
q20	507	330	319	319
q21	3069	2674	2697	2674
q22	352	274	284	274
Total cold run time: 46035 ms
Total hot run time: 40829 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4362	4392	4390	4390
q2	269	164	172	164
q3	3555	3536	3538	3536
q4	2397	2376	2383	2376
q5	5758	5763	5766	5763
q6	238	120	122	120
q7	2394	1874	1854	1854
q8	3538	3533	3537	3533
q9	9096	9036	9007	9007
q10	3914	3982	3989	3982
q11	502	391	414	391
q12	764	597	614	597
q13	4308	3541	3552	3541
q14	284	254	248	248
q15	574	510	518	510
q16	509	452	467	452
q17	1884	1835	1851	1835
q18	8791	8354	8391	8354
q19	1756	1781	1758	1758
q20	2251	1969	1931	1931
q21	6550	6203	6154	6154
q22	493	429	421	421
Total cold run time: 64187 ms
Total hot run time: 60917 ms

@Mryange
Copy link
Contributor Author

Mryange commented Dec 12, 2023

run pipelinex_p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 12, 2023
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@Gabriel39 Gabriel39 merged commit 15f5acf into apache:master Dec 12, 2023
xzj7019 pushed a commit to xzj7019/doris that referenced this pull request Dec 13, 2023
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. meta-change reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants