Add support for streaming delimited messages #529
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JoshuaLeivers
force-pushed
the
message-streaming
branch
3 times, most recently
from
04615a9 to
f300790
Compare
|
Sorry about the force-pushes, was just quickly fixing up the CI test fails before it could be reviewed properly, will avoid them from now on 🙂 |
|
This should finish addressing the message streaming part of #482 , would it be possible to get a review for this? |
JoshuaLeivers
force-pushed
the
message-streaming
branch
from
e3a6182 to
6a32f2a
Compare
|
Just rebased onto current |
This allows developers to easily dump and load multiple messages from a stream in a way that is compatible with official protobuf implementations (such as Java's `MessageLite#writeDelimitedTo(...)`).
These tests stream data such as messages to output files, have a Java binary read them and then write them back using the `protobuf-java` functions, and then read them back in on the Python side to check that the returned data is as expected. This checks that the official Java implementation (and so any other matching implementations) can properly parse outputs from Betterproto, and vice-versa, ensuring compatibility in these functions between the two.
JoshuaLeivers
force-pushed
the
message-streaming
branch
from
6a32f2a to
2d682a9
Compare
|
Had to re-sign the commits |
|
Looks reasonable to me, thanks for this work |
cetanu
approved these changes
Oct 16, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This allows developers to easily dump and load multiple messages
from a stream in a way that is compatible with official
protobuf implementations (such as Java's
MessageLite#writeDelimitedTo(...)).
This also adds compatibility tests between the official Java Protobuf
implementation and the new and existing streaming functions in
Betterproto. These tests stream data such as messages to output
files, have a Java binary read them and then write them back using
the protobuf-java functions, and then read them back in on the
Python side to check that the returned data is as expected. This
checks that the official Java implementation (and so any other
matching implementations) can properly parse outputs from
Betterproto, and vice-versa, ensuring compatibility in these
functions between the two.
The Java binary is built on the fly as part of a Pytest fixture, and
these tests are skipped if any necessary tools (i.e. Java or Maven)
are missing from the host machine, or if the build fails (e.g. due
to an incompatible Java version).