Storage format specification improvements 2/N #5329

teo-tsirpanis · 2024-09-30T23:53:01Z

Continuing #5205, this PR goes through each format version and mentions the changes to the main specification document. It also documents structures like __coords.tdb and legacy fragment metadata, and fixes any defects encountered in the meantime.

TYPE: FORMAT
DESC: The storage format specification was updated to document format changes of previous versions throughout the main document.

… and 17.

… to 11.

… to 6.

…o changes) and 2.

Specify which filters are used for each file, and break up a long bullet point.

format_spec/current_domain.md

format_spec/array_schema.md

teo-tsirpanis · 2024-10-07T17:00:36Z

format_spec/array_format_history.md

@@ -154,7 +153,7 @@ Introduced in TileDB 1.6
    * The [footer](./fragment.md#footer) and [R-Tree](./fragment.md#r-tree) structures were added.
    * The _Bounding coords_ field was removed.
    * The _MBRs_ field was removed. MBRs are now stored in the R-Tree.
-    * Structures other than the footer like tile offsets, sizes and metadata are wrapped in their own generic tiles. This allows loading them lazily and in parallel.
+    * Tile offsets and sizes are wrapped in their own generic tiles. This allows loading them lazily and in parallel.


Tile metadata did not exist back then.

teo-tsirpanis · 2024-10-07T17:02:21Z

format_spec/array_file_hierarchy.md

+  * Inside the same commits folder, any number of [consolidated commits files](./consolidated_commits_file.md) of the form [`<timestamped_name>`](./timestamped_name.md)`.con`.
+  * Inside the same commits folder, any number of [ignore files](./ignore_file.md) of the form [`<timestamped_name>`](./timestamped_name.md)`.ign`.


Consolidated commit files should have been gated behind a format version but they aren't.

teo-tsirpanis · 2024-10-08T22:57:03Z

format_spec/tile.md

@@ -35,6 +35,8 @@ The “chunk filtered data” bytes contain the final bytes of the chunk after b

 Internally, any filter in a filter pipeline produces two arrays of data as output: a metadata byte array and a filtered data byte array. Additionally, these output byte arrays can be arbitrarily separated into “parts” by any filter. Typically, when a next filter receives the output of the previous filter as its input, it will filter each “part” independently.

+_New in version 11_ Cells of arrays are not split across chunks within a tile.


I'm still not sure if this is format-breaking or not. There was some discussion in #5205 (comment).

Supporting splitting cells across chunks would enable some interesting optimizations. Consider for example storing two strings larger than the chunk size and one tiny string in between. They will be forced to be stored in three chunks. Maybe that's what we want for the RLE/dictionary filters, but then we can amend it to say "Cells of arrays are not split across chunks within a tile when using the RLE or dictionary encoding filter".

teo-tsirpanis · 2024-10-09T11:48:26Z

format_spec/array_format_history.md

@@ -72,7 +72,7 @@ Introduced in TileDB 2.10

 Introduced in TileDB 2.9

-* The [dictionary filter](./filters/dictionary_encoding.md) was added.


Filters are not "added" in a version so we have to reword this.

teo-tsirpanis · 2024-10-09T11:52:03Z

format_spec/filter_pipeline.md

@@ -80,6 +80,9 @@ For the `TILEDB_FILTER_DELTA` and `TILEDB_FILTER_DOUBLE_DELTA` compression filte
 | Compression level | `int32_t` | Ignored |
 | Reinterpret datatype | `uint8_t` | Type to reinterpret data prior to compression. |

+> [!NOTE]
+> Prior to version 20, the _Reinterpret datatype_ field was not present for the double delta filter. Also prior to version 19, the same field was not present for the delta filter.


The delta filter did not "exist" prior to version 19, but as said above filters are not introduced in specific format versions so we are specific that the filter option was introduced.

Would it maybe be better to phrase this like this instead:

The Reinterpret datatype field was added to the delta filter in version 19 and the double delta filter in version 20.

My preference is to document the latest version in the main specification¹ and mention behavior in prior versions in notes like this. In other words the spec has been consistently using "Prior to version X, Y did not exist", instead of "Y was added in version X".

Footnotes

An exception is simple additions and removals where I used the "New/Removed in version **" phrasing. Adding a "New in version 19 for the delta filter and new in version 20 for the double delta filter" would be too much for an inline comment. ↩

nickvigilante

LGTM overall. Great work! Just a few suggestions.

nickvigilante · 2024-10-14T15:28:05Z

format_spec/array_file_hierarchy.md

-* Inside the same commit folder, any number of [consolidated commits files](./consolidated_commits_file.md) of the form [`<timestamped_name>`](./timestamped_name.md)`.con`.
-* Inside the same commit folder, any number of [ignore files](./ignore_file.md) of the form [`<timestamped_name>`](./timestamped_name.md)`.ign`.
-* Inside of a fragment metadata folder, any number of [consolidated fragment metadata files](./consolidated_fragment_metadata_file.md) of the form [`<timestamped_name>`](./timestamped_name.md)`.meta`.
+* Inside of a `__schema` folder, any number of [array schema files](./array_schema.md) [`<timestamped_name>`](./timestamped_name.md).


Suggested change

* Inside of a `__schema` folder, any number of [array schema files](./array_schema.md) [`<timestamped_name>`](./timestamped_name.md).

* Inside the `__schema` folder, any number of [array schema files](./array_schema.md) [`<timestamped_name>`](./timestamped_name.md).

There's only one __schema folder per array, as far as I know.

format_spec/array_file_hierarchy.md

nickvigilante · 2024-10-14T15:29:08Z

format_spec/array_file_hierarchy.md

+* Inside of a `__schema` folder, any number of [array schema files](./array_schema.md) [`<timestamped_name>`](./timestamped_name.md).
+  * Note: the name does _not_ include the format version.
+  * _New in version 20_ Inside of the schema folder, an enumerations folder `__enumerations`.
+* Inside of a `__fragments` folder, any number of [fragment folders](./fragment.md) [`<timestamped_name>`](./timestamped_name.md).


Suggested change

* Inside of a `__fragments` folder, any number of [fragment folders](./fragment.md) [`<timestamped_name>`](./timestamped_name.md).

* Inside the `__fragments` folder, any number of [fragment folders](./fragment.md) [`<timestamped_name>`](./timestamped_name.md).

Using "the" might imply that the existence of the folder is required. There cannot be empty folders in cloud object storage which means that an array with no fragments yet written will not have a __fragments folder.

nickvigilante · 2024-10-14T15:31:18Z

format_spec/array_file_hierarchy.md

+  * Note: the name does _not_ include the format version.
+  * _New in version 20_ Inside of the schema folder, an enumerations folder `__enumerations`.
+* Inside of a `__fragments` folder, any number of [fragment folders](./fragment.md) [`<timestamped_name>`](./timestamped_name.md).
+* _New in version 12_ Inside of a `__commits` folder, an empty file [`<timestamped_name>`](./timestamped_name.md)`.wrt` associated with every fragment folder [`<timestamped_name>`](./timestamped_name.md), where [`<timestamped_name>`](./timestamped_name.md) is common for the folder and the WRT file. This is used to indicate that fragment [`<timestamped_name>`](./timestamped_name.md) has been *committed* (i.e., its write process finished successfully) and it is ready for use by TileDB. If the WRT file does not exist, the corresponding fragment folder is ignored by TileDB during the reads.


Suggested change

* _New in version 12_ Inside of a `__commits` folder, an empty file [`<timestamped_name>`](./timestamped_name.md)`.wrt` associated with every fragment folder [`<timestamped_name>`](./timestamped_name.md), where [`<timestamped_name>`](./timestamped_name.md) is common for the folder and the WRT file. This is used to indicate that fragment [`<timestamped_name>`](./timestamped_name.md) has been *committed* (i.e., its write process finished successfully) and it is ready for use by TileDB. If the WRT file does not exist, the corresponding fragment folder is ignored by TileDB during the reads.

* _New in version 12_ Inside the `__commits` folder lives an empty file [`<timestamped_name>`](./timestamped_name.md)`.wrt` associated with every fragment folder [`<timestamped_name>`](./timestamped_name.md), where [`<timestamped_name>`](./timestamped_name.md) is common for the folder and the WRT file. This is used to indicate that fragment [`<timestamped_name>`](./timestamped_name.md) has been *committed* (i.e., its write process finished successfully) and it is ready for use by TileDB. If the WRT file does not exist, the corresponding fragment folder is ignored by TileDB during the reads.

nickvigilante · 2024-10-14T15:33:04Z

format_spec/array_file_hierarchy.md

We should sort the file hierarchy updates by version (e.g., 20, followed by 18, followed by 16, followed by 12), similar to how format_spec/array_format_history.md is constructed.

Sorted by version (with unversioned items being put first because they are the most important) and grouped by directory hierarchy. I also reworded and simplified some items.

format_spec/array_schema.md

format_spec/current_domain.md

nickvigilante · 2024-10-14T15:40:02Z

format_spec/filter_pipeline.md

@@ -80,6 +80,9 @@ For the `TILEDB_FILTER_DELTA` and `TILEDB_FILTER_DOUBLE_DELTA` compression filte
 | Compression level | `int32_t` | Ignored |
 | Reinterpret datatype | `uint8_t` | Type to reinterpret data prior to compression. |

+> [!NOTE]
+> Prior to version 20, the _Reinterpret datatype_ field was not present for the double delta filter. Also prior to version 19, the same field was not present for the delta filter.


Would it maybe be better to phrase this like this instead:

The Reinterpret datatype field was added to the delta filter in version 19 and the double delta filter in version 20.

format_spec/fragment.md

Co-authored-by: Nick Vigilante <nickvigilante@users.noreply.github.com>

davisp

+1

teo-tsirpanis force-pushed the teo/format-spec-updates branch from 21b77df to 856bcd0 Compare October 4, 2024 16:56

teo-tsirpanis added 14 commits October 9, 2024 14:39

Fix typos.

d01e956

Footer length is always included in the latest format version.

574aee5

Cover changes in the storage format specification between versions 22…

9927daf

… and 17.

Move current domain spec to array schema.

8d10390

Cover changes in the storage format specification between versions 16…

b439208

… to 11.

Add clarification.

fd8002b

Cover changes in the storage format specification between versions 10…

477ecd6

… to 6.

Cover changes in the storage format specification in versions 5, 4 (n…

7463d3f

…o changes) and 2.

Cover changes in the storage format specification in version 3.

ff806c5

Specify current domain version.

bedf0e9

Clearly state that the main format version numbers are for arrays.

d1788c7

Fix omission.

0ee6375

Improve fragment documentation.

ae4fd0c

Specify which filters are used for each file, and break up a long bullet point.

More fixes.

b45e03f

teo-tsirpanis force-pushed the teo/format-spec-updates branch from 90a08e5 to b45e03f Compare October 9, 2024 11:39

teo-tsirpanis commented Oct 9, 2024

View reviewed changes

teo-tsirpanis marked this pull request as ready for review October 9, 2024 12:02

teo-tsirpanis requested review from ihnorton and a team October 9, 2024 12:02

nickvigilante approved these changes Oct 14, 2024

View reviewed changes

Address PR feedback.

8db322c

Co-authored-by: Nick Vigilante <nickvigilante@users.noreply.github.com>

ihnorton requested a review from davisp October 17, 2024 14:29

davisp approved these changes Oct 21, 2024

View reviewed changes

teo-tsirpanis merged commit 6564bd7 into dev Oct 25, 2024
4 checks passed

teo-tsirpanis deleted the teo/format-spec-updates branch October 25, 2024 13:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Storage format specification improvements 2/N #5329

Storage format specification improvements 2/N #5329

teo-tsirpanis commented Sep 30, 2024 •

edited

Loading

teo-tsirpanis Oct 7, 2024

teo-tsirpanis Oct 7, 2024

teo-tsirpanis Oct 8, 2024

teo-tsirpanis Oct 9, 2024

teo-tsirpanis Oct 9, 2024

nickvigilante Oct 14, 2024

teo-tsirpanis Oct 14, 2024 •

edited

Loading

nickvigilante left a comment

nickvigilante Oct 14, 2024

nickvigilante Oct 14, 2024

teo-tsirpanis Oct 14, 2024

nickvigilante Oct 14, 2024

nickvigilante Oct 14, 2024

teo-tsirpanis Oct 14, 2024 •

edited

Loading

nickvigilante Oct 14, 2024

davisp left a comment

		* Inside the same commits folder, any number of [consolidated commits files](./consolidated_commits_file.md) of the form [`<timestamped_name>`](./timestamped_name.md)`.con`.
		* Inside the same commits folder, any number of [ignore files](./ignore_file.md) of the form [`<timestamped_name>`](./timestamped_name.md)`.ign`.

		@@ -35,6 +35,8 @@ The “chunk filtered data” bytes contain the final bytes of the chunk after b

		Internally, any filter in a filter pipeline produces two arrays of data as output: a metadata byte array and a filtered data byte array. Additionally, these output byte arrays can be arbitrarily separated into “parts” by any filter. Typically, when a next filter receives the output of the previous filter as its input, it will filter each “part” independently.

		_New in version 11_ Cells of arrays are not split across chunks within a tile.

		@@ -72,7 +72,7 @@ Introduced in TileDB 2.10

		Introduced in TileDB 2.9

		* The [dictionary filter](./filters/dictionary_encoding.md) was added.

	* Inside of a `__schema` folder, any number of [array schema files](./array_schema.md) [`<timestamped_name>`](./timestamped_name.md).
	* Inside the `__schema` folder, any number of [array schema files](./array_schema.md) [`<timestamped_name>`](./timestamped_name.md).

	* Inside of a `__fragments` folder, any number of [fragment folders](./fragment.md) [`<timestamped_name>`](./timestamped_name.md).
	* Inside the `__fragments` folder, any number of [fragment folders](./fragment.md) [`<timestamped_name>`](./timestamped_name.md).

Storage format specification improvements 2/N #5329

Storage format specification improvements 2/N #5329

Conversation

teo-tsirpanis commented Sep 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teo-tsirpanis Oct 14, 2024 • edited Loading

Choose a reason for hiding this comment

Footnotes

nickvigilante left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teo-tsirpanis Oct 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davisp left a comment

Choose a reason for hiding this comment

teo-tsirpanis commented Sep 30, 2024 •

edited

Loading

teo-tsirpanis Oct 14, 2024 •

edited

Loading

teo-tsirpanis Oct 14, 2024 •

edited

Loading