-
Notifications
You must be signed in to change notification settings - Fork 33
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
consolidation followed by vacuum can corrupt array #1885
Comments
Thanks for reporting this @bkmartinjr. I will look at the issue soon! |
KiterLuc
added a commit
to TileDB-Inc/TileDB
that referenced
this issue
Jan 3, 2024
The following issue TileDB-Inc/TileDB-Py#1885 reported that running consolidation in this order invalidates an array: - consolidate fragments - consolidate commits - vacuum fragments - vacuum commits This fixes two issues: - We sometimes would not delete the commit files when vacuuming fragments. - We would vacuum an ignore file when a consolidated commits file still needed it. --- TYPE: BUG DESC: Fix out of order consolidation.
Fix in progress: TileDB-Inc/TileDB#4597. |
KiterLuc
added a commit
to TileDB-Inc/TileDB
that referenced
this issue
Jan 5, 2024
The following issue TileDB-Inc/TileDB-Py#1885 reported that running consolidation in this order invalidates an array: - consolidate fragments - consolidate commits - vacuum fragments - vacuum commits This fixes two issues: - We sometimes would not delete the commit files when vacuuming fragments. - We would vacuum an ignore file when a consolidated commits file still needed it. --- TYPE: BUG DESC: Fix out of order consolidation.
Hi @bkmartinjr, could you try this again on 0.26.0? This should be fixed now. |
Confirmed that it works in my hands. |
github-actions bot
pushed a commit
to TileDB-Inc/TileDB
that referenced
this issue
Apr 19, 2024
The following issue TileDB-Inc/TileDB-Py#1885 reported that running consolidation in this order invalidates an array: - consolidate fragments - consolidate commits - vacuum fragments - vacuum commits This fixes two issues: - We sometimes would not delete the commit files when vacuuming fragments. - We would vacuum an ignore file when a consolidated commits file still needed it. --- TYPE: BUG DESC: Fix out of order consolidation. (cherry picked from commit 65d1ce4)
Shelnutt2
pushed a commit
to TileDB-Inc/TileDB
that referenced
this issue
Apr 22, 2024
The following issue TileDB-Inc/TileDB-Py#1885 reported that running consolidation in this order invalidates an array: - consolidate fragments - consolidate commits - vacuum fragments - vacuum commits This fixes two issues: - We sometimes would not delete the commit files when vacuuming fragments. - We would vacuum an ignore file when a consolidated commits file still needed it. --- TYPE: BUG DESC: Fix out of order consolidation. (cherry picked from commit 65d1ce4)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The order of consolidate/vacuum operations seems to create unreadable arrays in some cases. If I create a multi-fragment array, and then:
then the array is no longer readable.
Instead, if I change the order of operations to the following, all is well:
Based on prior advice, I was under the impression that either order would work fine. In any case, I would not expect an unusable array after any operation.
Reproducible test case below, run on Linux TileDB-Py version 0.24.0.
When I run it, I get the following output:
Code:
The text was updated successfully, but these errors were encountered: