Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RoundTripEnvelope should use a more compact data format #5

Open
muse254 opened this issue Jan 14, 2025 · 4 comments
Open

RoundTripEnvelope should use a more compact data format #5

muse254 opened this issue Jan 14, 2025 · 4 comments
Labels
question Further information is requested

Comments

@muse254
Copy link
Collaborator

muse254 commented Jan 14, 2025

Problem

If you compare the encoded data for file assets that is sent encoded to JSON, we have a scenario where ~300kb file asset becomes a ~2MB json payload.
A good approach would be to create or use custom binary encoding formats to optimise for size. This would be perfectly backwards compatible with the current API we have in place.
More on this: https://stackoverflow.com/questions/49096721/json-transfer-vs-binary-transfer

@stravid87
Copy link

stravid87 commented Jan 14, 2025

Honestly, I have always imagined that the interceptor would look something like this:

image

With layers being added / removed as necessary.
"A good approach would be to create or use custom binary encoding formats to optimize for size."
I agree so long as we can find an out of the box solution. For obvious reasons, creating our own -- or even creating an implementation of a well done spec like grpc -- is probably too much.
Instead of encoding, maybe we just consider compression?
https://docs.rs/brotli/latest/brotli/

@muse254
Copy link
Collaborator Author

muse254 commented Jan 14, 2025

We added compression in this PR, compression, I should have mentioned that @stravid87.

@muse254
Copy link
Collaborator Author

muse254 commented Jan 14, 2025

Let me interest you in one such implementation we can plug and play: https://github.com/bincode-org/bincode

@muse254
Copy link
Collaborator Author

muse254 commented Jan 19, 2025

Followup on this issue, we hit an improvement ratio of 0.9998 for a 5MB file asset sample when using bincode compared to JSON bytes.
JSON doesn't seem to be the problem

Leaving the issue open for later investigation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants