Skip to content

Commit

Permalink
GH-486: Variant object shredding without field shredding (#487)
Browse files Browse the repository at this point in the history
  • Loading branch information
gszadovszky authored Mar 6, 2025
1 parent ae5b9d7 commit 3d8855a
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions VariantShredding.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ Readers can assume that a value is not an object if `typed_value` is null and th
Each shredded field in the `typed_value` group is represented as a required group that contains optional `value` and `typed_value` fields.
The `value` field stores the value as Variant-encoded `binary` when the `typed_value` cannot represent the field.
This layout enables readers to skip data based on the field statistics for `value` and `typed_value`.
The `typed_value` field may be omitted when not shredding fields as a specific type.

The `value` column of a partially shredded object must never contain fields represented by the Parquet columns in `typed_value` (shredded fields).
Readers may always assume that data is written correctly and that shredded fields in `typed_value` are not present in `value`.
Expand Down

0 comments on commit 3d8855a

Please sign in to comment.