Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Union Layout Should Not Support Separate Validity Mask #1590

Closed
tustvold opened this issue Apr 19, 2022 · 1 comment · Fixed by #1589
Closed

Union Layout Should Not Support Separate Validity Mask #1590

tustvold opened this issue Apr 19, 2022 · 1 comment · Fixed by #1589
Assignees
@tustvold
Labels
bug

Comments

@tustvold
Copy link
Contributor

Describe the bug

Unlike other data types, unions do not have their own validity bitmap. Instead, the nullness of each slot is determined exclusively by the child arrays which are composed to create the union.

From https://arrow.apache.org/docs/format/Columnar.html#union-layout

To Reproduce

We currently support this, which aside from adding a lot of complexity, is incorrect

Expected behavior

We shouldn't support something not in the arrow specification
@tustvold tustvold added the bug label Apr 19, 2022
@tustvold tustvold changed the title Union Layout Should Not Support Validity Mask Union Layout Should Not Support Separate Validity Mask Apr 19, 2022
@tustvold
Copy link
Contributor Author

This does come with a rather strange implication, that nulls are typed which is a little odd...

@tustvold tustvold self-assigned this Apr 19, 2022
tustvold added a commit to tustvold/arrow-rs that referenced this issue Apr 20, 2022
@tustvold
Simplify null masking in equality comparisons
95b2ed3 
Various UnionArray fixes (apache#1598) (apache#1596) (apache#1591) (apache#1590)

Fix handling of null masks in ArrayData equality (apache#1599)
@tustvold tustvold mentioned this issue Apr 20, 2022
Merged
alamb pushed a commit that referenced this issue Apr 28, 2022
@tustvold
Fix Null Mask Handling in ArrayData And UnionArray (#1589)
37085d2 
* Fix ListArray and StructArray equality (#626)

* Simplify null masking in equality comparisons

Various UnionArray fixes (#1598) (#1596) (#1591) (#1590)

Fix handling of null masks in ArrayData equality (#1599)

* Miscellaneous fixes

* Fix structure null equality

* Review feedback
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tustvold tustvold

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix Null Mask Handling in ArrayData, UnionArray, and MapArray tustvold/arrow-rs
1 participant
@tustvold