feat: Add IPC stream writing #571
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
bkietz
commented
Aug 1, 2024
•
bkietz
commented
- adds ArrowIpcArrayStreamWriter along with Init() and Reset() methods
- embeds an Encoder and an OutputStream
- adds WriteSchema() and WriteArrayView() methods which encode then write a schema or array
- adds WriteArrayStream() method which serializes an entire ArrowArrayStream, finishing with an explicit EOS
- non-blocking IO is not currently supported
bkietz
force-pushed
the
ipc-stream-writing
branch
3 times, most recently
from
47bb06a
to
708574e
Compare
paleolimbot
reviewed
Aug 3, 2024
bkietz
force-pushed
the
ipc-stream-writing
branch
from
708574e
to
5973471
Compare
bkietz
force-pushed
the
ipc-stream-writing
branch
from
5973471
to
0cc7f7c
Compare
paleolimbot
reviewed
Aug 7, 2024
src/nanoarrow/nanoarrow_ipc.h
Outdated
| /// takes ownership of the output byte stream and the encoder, and the caller is | ||
| /// responsible for releasing the writer by calling ArrowIpcWriterReset(). | ||
| ArrowErrorCode ArrowIpcWriterInit(struct ArrowIpcWriter* writer, | ||
| struct ArrowIpcEncoder* encoder, |
There was a problem hiding this comment.
Would it make sense to initialize the encoder internally here instead of force the caller to initialize one and then immediately move it?
There was a problem hiding this comment.
I was thinking that the keeping the encoder's initialization separate allows us to avoid duplicating options for encoder. However we can probably just give configuration access to the stream's internal encoder when that becomes necessary.
|
Valgrind is complaining about an invalid read. I can reproduce this locally and I get AFAICT from preliminary debugging flatcc's usage of realloc() is not incorrect, and Valgrind doesn't complain about the same test compiled using clang. ASAN also doesn't produce any error here. In particular I don't get any segfaults or Valgrind failures when replacing realloc with an equivalent malloc+memcpy+free (which should expose the common realloc error of not accounting for the memory block moving). Maybe this should just be handled by a new suppression? Alternatively flatcc supports straightforward configuration of whether realloc will be used ( |
paleolimbot
approved these changes
Aug 8, 2024
There was a problem hiding this comment.
From a tiny bit of reading about this, Valgrind's realloc() implementation moves every block and frees the original allocation. Debugging this may take a while...maybe add the suppression with a follow-up tagged with a milestone for the next version? It may be easier to generate more cases where this occurs when there are R or Python bindings, too.
|
Now valgrind is complaining about protobuf singletons which are not free'd before exit: I'll add a suppression for that as well. |
|
Erg. (I wonder how protobuf sneaked into our "minimal" C++ build for CI!) |
|
It's arrow is linked to protobuf, and the tests link to arrow. What's odd to me is this wasn't in arrow's own suppressions file |
|
Okay, that's fun: since we are linked to protobuf, we have inherited responsibility to call ShutdownProtobufLibrary in order to clean up these singletons. That's not something libarrow does or should do, because the presumption in libarrow is anyone who uses protobuf features will want control over when protobuf gets shut down. |
Gumix
reviewed
Aug 10, 2024
|
|
||
| NANOARROW_RETURN_NOT_OK(ArrowIpcEncoderEncodeSimpleRecordBatch( | ||
| &private->encoder, in, &private->body_buffer, error)); | ||
|
|
There was a problem hiding this comment.
Hello,
Shouldn't private->buffer be finalized here? Otherwise the header of a message isn't written to the output stream.
Like this:
NANOARROW_RETURN_NOT_OK_WITH_ERROR(
ArrowIpcEncoderFinalizeBuffer(&private->encoder, /*encapsulate=*/1,
&private->buffer),
error);There was a problem hiding this comment.
You know... it really should. Why isn't this failing tests?
There was a problem hiding this comment.
I have the same question :) When I tried to use this API, I noticed the absence of the header.
There was a problem hiding this comment.
Okay... I'll get a fix as soon as I write a failing test. Thanks!
bkietz
added a commit
that referenced
this pull request
Aug 12, 2024
- As noted #571 (comment) RecordBatch Messages were not being written during IPC streaming. This has been corrected - Testing uses one more round trip case where we have arrow C++ read from a stream written by nanoarrow. - During testing it also became apparent that encapsulated messages were not always stored with the correct size; metadata_size should include padding (but not the continuation or size itself) https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-format
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.