Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom DatumReader/DatumWriter #445

Open
bplommer opened this issue May 5, 2022 · 0 comments
Open

Custom DatumReader/DatumWriter #445

bplommer opened this issue May 5, 2022 · 0 comments

Comments

@bplommer
Copy link
Member

bplommer commented May 5, 2022

The current implementation that converts to and from the Java GenericDatumReader/GenericDatumWriter's representation is unsatisfactory for reasons that have been previously discussed - instead it's been suggested that we reimplement Avro using scodec. Instead I think the best way forward would be to continue using the Java Decoder and Encoder (which handle the low-level mechanics of converting between byte streams and JVM primitives, while abstracting over the binary and json encodings) but have our own implementations of DatumReader and DatumWriter, which orchestrate the Decoder/Encoder operations to work with complete data structures. That gives us complete control of the representations we work with and avoids any extra indirection at runtime, while giving us the benefit of encoders/decoders that are likely to be better tested and more performant than anything we write ourselves, and will interoperate smoothly with the Confluent Kafka serdes.

Easiest to do after Codec has been converted to a data structure per #437 - we can then add functionality, separate from the Codec, to compile it (together with an optional schema) to a DatumReader or DatumWriter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant