ARROW-94: [Format] Expand list example to clarify null vs empty list #58

emkornfield · 2016-04-09T07:00:32Z

WIP to make sure what I've done so far looks good. Per discussion on the JIRA item I started conversion of examples images to "text diagrams", but I wanted to get feedback to see if this is actually desirable (and if the way I'm approaching it is desirable). The remaining diagrams are for unions which I can convert if the existing changes look OK to others (although I think the Union diagrams are are pretty reasonable/compact).

This change also includes some other minor cleanup, as well as including a statement about endianness per the discussion on the mailing list.

Rendered markdown can be seen at: https://github.com/emkornfield/arrow/blob/emk_doc_fixes_PR3/format/Layout.md

…or struct

wesm · 2016-04-09T13:47:08Z

format/Layout.md


 While a struct does not have physical storage for each of its semantic slots
 (i.e. each scalar C-like struct), an entire struct slot can be set to null via
 the null bitmap. Any of the child field arrays can have null values according
 to their respective independent null bitmaps.
+In the example above, the child arrays have a valid entries for the null struct
+but are 'hidden' from the consumer by the parent array's null bitmap.


As I recall there have been some questions around whether the child arrays' bitmaps must necessarily be set to null if the parent struct slot is null. I think the answer is "no" (and, in fact, you could combine a set of immutable constructed-elsewhere arrays with a bitmap that you layer on top to "null out" those other values), so having to twiddle other bitmaps would be onerous and ultimately not that useful.

If you agree perhaps we can spell this out here also specifically.

I do agree and I thought I read this elsewhere in the spec, but sounds like it is worth confirming the mailing list?

There is this statement already: "Any of the child field arrays can have null values according to their respective independent null bitmaps." I don't think it was ever a controversial point but rather just unclear -- feel free to raise it on the mailing list if you like, though.

Rereading the spec, I agree. I will update the main text accordingly.

wesm · 2016-04-09T13:51:44Z

This looks great to me -- I think using text diagrams going forward will be more sustainable and clear to external readers, though hopefully we are close to finalizing some of the lingering format details!

…ome typos

drankye · 2016-04-10T12:27:24Z

format/Layout.md

@@ -198,7 +308,8 @@ types (which can all be distinct), called its fields.
 Typically the fields have names, but the names and their types are part of the
 type metadata, not the physical memory layout.

-A struct does not have any additional allocated physical storage.
+A struct array does not have any additional allocated physical storage for its values.
+A struct array must still have an allocated null bitmap, if it has one or more null values.



I thought a struct array will always have the null bitmap, regardless being of any null entry or not?

It isn't required to have the memory allocated if there are no nulls, like the other array types.

Got it, thx. I'm not sure about how these types are used. When creating the object of such type, it must be known before if any null exists by scanning the data to put into it?

Consider the following dataset:

data = [ {foo: 1, bar: null}, null, null, {foo: 0, bar: 5} ]

wesm · 2016-04-13T11:55:01Z

Do you want to also delete the existing graphical diagrams as part of this patch?

emkornfield · 2016-04-14T08:02:46Z

Might as well just pushed another commit.

wesm · 2016-04-14T17:23:44Z

+1. Merging this. If there is something causing lingering controversy from this patch I encourage everyone to continue discussion on the mailing list so we can make further decisions there where relevant.

…related refactoring This also restores passing on user's `CMAKE_CXX_FLAGS`, which had unfortunately led some compiler warnings to creep into our build. Author: Wes McKinney <wesm@apache.org> Closes apache#58 from wesm/PARQUET-457 and squashes the following commits: 4bf12ed [Wes McKinney] * SerializeThriftMsg now writes into an OutputStream. * Refactor page serialization in advance of compression tests * Test compression roundtrip on random bytes for snappy and gzip * Trying LZO compression results in ParquetException * Don't lose user's CMAKE_CXX_FLAGS * Remove Travis CI directory caching for now * Fix gzip memory leak if you do not call inflateEnd, deflateEnd

…related refactoring This also restores passing on user's `CMAKE_CXX_FLAGS`, which had unfortunately led some compiler warnings to creep into our build. Author: Wes McKinney <wesm@apache.org> Closes apache#58 from wesm/PARQUET-457 and squashes the following commits: 4bf12ed [Wes McKinney] * SerializeThriftMsg now writes into an OutputStream. * Refactor page serialization in advance of compression tests * Test compression roundtrip on random bytes for snappy and gzip * Trying LZO compression results in ParquetException * Don't lose user's CMAKE_CXX_FLAGS * Remove Travis CI directory caching for now * Fix gzip memory leak if you do not call inflateEnd, deflateEnd Change-Id: I44a58ef2d22f8e5064d198d0abeecde7ba4de3cb

Add List input and output types for Gandiva functions. Add new reference implementations for array_contains and array_remove, tested via integration with Dremio. int32, int64, double and float list types have been tested. Support List types in function specification and llvm code generation. Pass back function type information through the expression registry. See 1p here: https://docs.google.com/document/d/1exwXdUUnk5FqZLzVZyTdhqgwxTk0u9bL54aLVNM5Tas/edit

emkornfield added 2 commits April 7, 2016 04:57

cleanup magic quotes and clarify/fix some minor points

590e4a7

replace diagrams with physical layouts, clarify memory requirements f…

0f23052

…or struct

wesm reviewed Apr 9, 2016
View reviewed changes

emkornfield added 2 commits April 10, 2016 04:36

clarify requirements of child types, finish replacing diagrams, fix s…

7dda5d5

…ome typos

change list offset array to offset buffer

b7aa7ea

emkornfield changed the title ~~[WIP] ARROW-94: [Format] Expand list example to clarify null vs empty list~~ ARROW-94: [Format] Expand list example to clarify null vs empty list Apr 10, 2016

fix some alignment, and one last offset array to buffer conversion

69e1a78

drankye reviewed Apr 10, 2016
View reviewed changes

emkornfield added 2 commits April 10, 2016 18:47

fix some off by one bugs

5550a78

a few more consistency fixes

cab6f87

remove png diagrams that are no longer used

00b99ef

asfgit closed this in 37f7271 Apr 14, 2016

emkornfield deleted the emk_doc_fixes_PR3 branch April 19, 2016 07:36

zhouyuan pushed a commit to zhouyuan/arrow that referenced this pull request Dec 22, 2021

Update Intel Security.md (apache#58)

53dcb36

zhztheplayer pushed a commit to zhztheplayer/arrow-1 that referenced this pull request Feb 8, 2022

Update Intel Security.md (apache#58)

4d0871b

zhztheplayer pushed a commit to zhztheplayer/arrow-1 that referenced this pull request Mar 3, 2022

Update Intel Security.md (apache#58)

10ef761

rui-mo pushed a commit to rui-mo/arrow-1 that referenced this pull request Mar 23, 2022

Update Intel Security.md (apache#58)

bb2b95a

paleolimbot mentioned this pull request Jan 28, 2023

[R] Crash on MacOS (x86) when running tests with homebrew apache-arrow also installed #33903

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-94: [Format] Expand list example to clarify null vs empty list #58

ARROW-94: [Format] Expand list example to clarify null vs empty list #58

emkornfield commented Apr 9, 2016

wesm Apr 9, 2016

emkornfield Apr 9, 2016

wesm Apr 9, 2016

emkornfield Apr 9, 2016

wesm commented Apr 9, 2016

drankye Apr 10, 2016

wesm Apr 10, 2016

drankye Apr 10, 2016

wesm Apr 10, 2016

wesm commented Apr 13, 2016

emkornfield commented Apr 14, 2016

wesm commented Apr 14, 2016

ARROW-94: [Format] Expand list example to clarify null vs empty list #58

ARROW-94: [Format] Expand list example to clarify null vs empty list #58

Conversation

emkornfield commented Apr 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wesm commented Apr 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wesm commented Apr 13, 2016

emkornfield commented Apr 14, 2016

wesm commented Apr 14, 2016