Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: optimize table t compact segment only #8247

Closed
Tracked by #7823
dantengsky opened this issue Oct 17, 2022 · 0 comments · Fixed by #8261
Closed
Tracked by #7823

Feature: optimize table t compact segment only #8247

dantengsky opened this issue Oct 17, 2022 · 0 comments · Fixed by #8261
Assignees
Labels
C-feature Category: feature

Comments

@dantengsky
Copy link
Member

dantengsky commented Oct 17, 2022

Summary

new option for optimizing table

 optimize table t compact [segment]

to compact table meta data only (mainly the segments). This operation is expected

  • to be much lighter than optimize table t compact,
  • and can be invoked concurrently while the same table is being appended.

There are the cases that a table is kept being appended will large data chunks(but not large enough to construct a compact segment -- at least 1000 blocks, for the default table options), and at a relatively high frequency(say several commits per sec); the blocks generated are mostly non-fragmented, but the segments are not compact (usually only contain 1 ~10 meta data of blocks).

the un-compact segments harm both the reading and the data ingestion of the table

  • for the former, reading lots of fragmented segments harms
  • for the latter, the size of snapshot metadata, and the time to write it down, increases,
@dantengsky dantengsky added the C-feature Category: feature label Oct 17, 2022
@dantengsky dantengsky self-assigned this Oct 17, 2022
@BohuTANG BohuTANG changed the title Feature: optimize table t compact meta only Feature: optimize table t compact segment only Oct 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-feature Category: feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant