consolidation followed by vacuum can corrupt array

[bkmartinjr]

The order of consolidate/vacuum operations seems to create unreadable arrays in some cases. If I create a multi-fragment array, and then:

1. consolidate fragments
2. consolidate commits
3. vacuum fragments
4. vacuum commits
   then the array is no longer readable.

Instead, if I change the order of operations to the following, all is well:

1. consolidate fragments
2. vacuum fragments
3. consolidate commits
4. vacuum commits

Based on prior advice, I was under the impression that either order would work fine. In any case, I would not expect an unusable array after any operation.

Reproducible test case below, run on Linux TileDB-Py version 0.24.0.

When I run it, I get the following output:

$ python tmp/consolvac.py 
{'array_schema_name': ('__1703625051952_1703625051952_e543e42260074ff3a84460285efb2389',
                       '__1703625051952_1703625051952_e543e42260074ff3a84460285efb2389'),
 'array_uri': 'foobar',
 'cell_num': (100, 100),
 'has_consolidated_metadata': (False, False),
 'nonempty_domain': (((0, 99), (759, 57697)), ((100, 199), (543, 56478))),
 'sparse': (True, True),
 'timestamp_range': ((1703625051969, 1703625051969),
                     (1703625051998, 1703625051998)),
 'to_vacuum': (),
 'unconsolidated_metadata_num': 2,
 'uri': ('file:///home/bruce/projects/cellxgene-census/foobar/__fragments/__1703625051969_1703625051969_33f43f9c71d04370a54331865ec608b7_20',
         'file:///home/bruce/projects/cellxgene-census/foobar/__fragments/__1703625051998_1703625051998_90b46f58c6fc4a448571b7f923029dab_20'),
 'version': (20, 20)}
Traceback (most recent call last):
  File "/home/bruce/projects/cellxgene-census/tmp/consolvac.py", line 67, in <module>
    main()
  File "/home/bruce/projects/cellxgene-census/tmp/consolvac.py", line 46, in main
    pprint(tiledb.array_fragments(uri))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bruce/projects/cellxgene-census/venv/lib/python3.11/site-packages/tiledb/highlevel.py", line 190, in array_fragments
    return tiledb.FragmentInfoList(uri, include_mbrs, ctx)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bruce/projects/cellxgene-census/venv/lib/python3.11/site-packages/tiledb/fragment.py", line 107, in __init__
    fi = PyFragmentInfo(self.array_uri, schema, include_mbrs, ctx)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tiledb.cc.TileDBError: [TileDB::IO] Error: Cannot get file size of '/home/bruce/projects/cellxgene-census/foobar/__fragments/__1703625051998_1703625051998_90b46f58c6fc4a448571b7f923029dab_20/__fragment_metadata.tdb'; No such file or directory (tiledb/fragment.cc:121)

Code:

import numpy as np
import tiledb
from pprint import pprint


def main():
    uri = "foobar"
    n_frags = 2
    frag_size = 100

    schema = tiledb.ArraySchema(
        domain=tiledb.Domain(
            *[
                tiledb.Dim(
                    name="soma_dim_0", domain=(0, 31469), tile=2048, dtype="int64"
                ),
                tiledb.Dim(
                    name="soma_dim_1", domain=(0, 57876), tile=2048, dtype="int64"
                ),
            ]
        ),
        attrs=[tiledb.Attr(name="soma_data", dtype="float32")],
        cell_order="row-major",
        tile_order="row-major",
        capacity=65536,
        sparse=True,
        allows_duplicates=True,
    )

    rng = np.random.default_rng()
    tiledb.SparseArray.create(uri, schema=schema)
    with tiledb.open(uri, mode="w") as A:
        for frag in range(n_frags):
            A[
                (
                    np.arange(
                        frag_size * frag, frag_size * frag + frag_size, dtype=np.int64
                    ),
                    rng.integers(0, 57876, size=frag_size),
                )
            ] = rng.random(size=frag_size, dtype=np.float32)

    pprint(tiledb.array_fragments(uri))

    _consolidate_array(uri, vacuum=True)
    pprint(tiledb.array_fragments(uri))


def _consolidate_array(uri: str, vacuum: bool) -> None:
    modes = ["fragments", "commits"]

    for mode in modes:
        tiledb.consolidate(
            uri,
            config=tiledb.Config({"sm.consolidation.mode": mode}),
        )

    if vacuum:
        for mode in modes:
            tiledb.vacuum(
                uri,
                config=tiledb.Config({"sm.vacuum.mode": mode}),
            )


if __name__ == "__main__":
    main()

[KiterLuc]

Thanks for reporting this @bkmartinjr. I will look at the issue soon!

[KiterLuc]

Fix in progress: TileDB-Inc/TileDB#4597.

[KiterLuc]

Hi @bkmartinjr, could you try this again on 0.26.0? This should be fixed now.

[bkmartinjr]

Confirmed that it works in my hands.