Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug fix: schema evolution fix-to-var reads followup. #5362

Merged
merged 1 commit into from
Nov 12, 2024

Conversation

bekadavis9
Copy link
Contributor

Schema evolution bug fix: Reads no longer fail after dropping a fixed attribute and adding it back as var-sized.

Followup to #5321 to address a previously-missed case caught by TileDB-Inc/TileDB-Py/#2083

[sc-55085]


TYPE: BUG
DESC: Schema evolution bug fix: Reads no longer fail after dropping a fixed attribute and adding it back as var-sized, part 2.

@kounelisagis
Copy link
Member

kounelisagis commented Oct 30, 2024

Just to make sure we don’t lose or forget (talked with @bekadavis9 about it).

If I change my test to only have a single argument:

def test_schema_evolution_drop_fixed_attribute_and_add_back_as_var_sized(tmp_path):
    ctx = tiledb.default_ctx()
    uri = str(tmp_path)
    attrs = [
        tiledb.Attr(name="a", dtype=np.int32),
    ]
    dims = [tiledb.Dim(domain=(1, 10), dtype=np.int32)]
    domain = tiledb.Domain(*dims)
    schema = tiledb.ArraySchema(domain=domain, attrs=attrs, sparse=False)
    tiledb.Array.create(uri, schema)

    data = np.arange(1, 11)
    with tiledb.open(uri, "w") as A:
        A[:] = {"a": data}

    se = tiledb.ArraySchemaEvolution(ctx)
    se.drop_attribute("a")
    se.array_evolve(uri)

    se = tiledb.ArraySchemaEvolution(ctx)
    newattr = tiledb.Attr("a", dtype="S", var=True)
    se.add_attribute(newattr)
    se.array_evolve(uri)

I am getting tiledb.cc.TileDBError: ArraySchema: Array schema check failed; No attributes provided coming from the last line.
I think this is not ideal behavior. I understand that ArraySchema should always have an argument, so I believe an error should be thrown when the attribute is dropped instead(?).

Copy link
Member

@kounelisagis kounelisagis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After updating the test in TileDB-Inc/TileDB-Py#2083, it seems we now have the expected behavior. After discussing with @bekadavis9, although the test I mentioned in my previous comment reveals a non-ideal behavior, addressing it would require a major rewrite of the SchemaEvolution class and is beyond the scope of this PR.

With that in mind, this looks good to me. Nice work!

@bekadavis9 bekadavis9 merged commit abca93c into dev Nov 12, 2024
64 checks passed
@bekadavis9 bekadavis9 deleted the rd/bugfix-schema_evolution_followup branch November 12, 2024 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants