feat: add Footer decoding #598
Conversation
bkietz
commented
Aug 27, 2024
bkietz
commented
- Adds ArrowIpcDecoderPeekFooter(), ArrowIpcDecoderVerifyFooter(), and ArrowIpcDecoderDecodeFooter()
- Uses these to read IPC files in the integration test executable
| NANOARROW_RETURN_NOT_OK(ArrowIpcDecoderVerifyFooter( | ||
| decoder.get(), {{bytes.data()}, static_cast<int64_t>(bytes.size())}, error)); | ||
| NANOARROW_RETURN_NOT_OK(ArrowIpcDecoderDecodeFooter( | ||
| decoder.get(), {{bytes.data()}, static_cast<int64_t>(bytes.size())}, error)); |
There was a problem hiding this comment.
Should verify and decode be the same function rather than separate functions?
There was a problem hiding this comment.
I'm following the pattern of schema decoding, where the two are separated. Verification checks offsets to ensure all flatbuffers are in bounds, while decode actually copies flatbuffers into nanoarrow data structures. I'd be fine consolidating them but that should be a follow up
There was a problem hiding this comment.
Definitely possible to consolidate them...I think I separated them initially in case somebody owned their whole pipeline and wanted to skip it as an optimization (which is probably not actually any faster, and probably nobody will want do do). For some flatbuffer uses this matters ( http://flatgeobuf.org/#why-am-i-not-getting-expected-performance-in-gdal ).
| NANOARROW_RETURN_NOT_OK( | ||
| ArrowIpcDecoderSetSchema(decoder.get(), &decoder->footer->schema, error)); | ||
| NANOARROW_RETURN_NOT_OK_WITH_ERROR( | ||
| ArrowIpcDecoderSetEndianness(decoder.get(), decoder->endianness), error); |
There was a problem hiding this comment.
Isn't endianness a flag in the schema?
There was a problem hiding this comment.
Endianness is a flag in the IPC schema message, but not in ArrowSchema which is what decoder->footer->schema is, so we need to set it up separately
| data.data.as_uint8 + data.size_bytes - footer_and_size_and_magic_size; | ||
|
|
||
| // Run flatbuffers verification | ||
| if (ns(Footer_verify_as_root(footer_data, decoder->header_size_bytes) != |
There was a problem hiding this comment.
flatcc generates symbols like org_apache_arrow_flatbuf_Footer_verify_as_root. ns() is a macro which prepends that namespace
