You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
to compact table meta data only (mainly the segments). This operation is expected
to be much lighter than optimize table t compact,
and can be invoked concurrently while the same table is being appended.
There are the cases that a table is kept being appended will large data chunks(but not large enough to construct a compact segment -- at least 1000 blocks, for the default table options), and at a relatively high frequency(say several commits per sec); the blocks generated are mostly non-fragmented, but the segments are not compact (usually only contain 1 ~10 meta data of blocks).
the un-compact segments harm both the reading and the data ingestion of the table
for the former, reading lots of fragmented segments harms
for the latter, the size of snapshot metadata, and the time to write it down, increases,
The text was updated successfully, but these errors were encountered:
Summary
new option for optimizing table
to compact table meta data only (mainly the segments). This operation is expected
optimize table t compact
,There are the cases that a table is kept being appended will large data chunks(but not large enough to construct a compact segment -- at least 1000 blocks, for the default table options), and at a relatively high frequency(say several commits per sec); the blocks generated are mostly non-fragmented, but the segments are not compact (usually only contain 1 ~10 meta data of blocks).
the un-compact segments harm both the reading and the data ingestion of the table
The text was updated successfully, but these errors were encountered: