Skip to content

improvement: move table compact to pipeline builder #7924

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Tracked by #7823
BohuTANG opened this issue Sep 27, 2022 · 3 comments
Closed
Tracked by #7823

improvement: move table compact to pipeline builder #7924

BohuTANG opened this issue Sep 27, 2022 · 3 comments
Assignees
Labels
C-improvement Category: improvement

Comments

@BohuTANG
Copy link
Member

BohuTANG commented Sep 27, 2022

Summary

Now the compact processor build is in a hard way:
https://github.com/datafuselabs/databend/blob/5ae594effc7b0f4766e5e680f1b709f5a179af2d/src/query/storages/fuse/src/operations/compact.rs#L70-L119

It would be great to move it to the pipeline_builder. When the pipeline builder meets a CompactPlan, it would add the CompactTransform to the pipeline builder.

@BohuTANG BohuTANG added the C-improvement Category: improvement label Sep 27, 2022
@BohuTANG
Copy link
Member Author

cc @zhyass

@zhyass zhyass self-assigned this Sep 29, 2022
@BohuTANG
Copy link
Member Author

The most cost of the compact is to read segments, now it's in parallel, it's fast, let's close this issue.

@BohuTANG BohuTANG reopened this Oct 17, 2022
@BohuTANG
Copy link
Member Author

BohuTANG commented Nov 9, 2022

Outdate, let's close.

@BohuTANG BohuTANG closed this as completed Nov 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-improvement Category: improvement
Projects
None yet
Development

No branches or pull requests

2 participants