Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support protobuf deserialization from Schema Registry and without a generated descriptor #36

Open
sauljabin opened this issue Jul 16, 2024 · 2 comments
Labels
blocked Depends on another issue feature request New feature

Comments

@sauljabin
Copy link
Owner

sauljabin commented Jul 16, 2024

It's necessary that confluent-kafka-python provides support for this. It has it for java.
Right now the DESCRIPTOR is needed https://github.com/confluentinc/confluent-kafka-python/blob/master/src/confluent_kafka/schema_registry/protobuf.py#L482.

  1. Extract the MessageIndexes https://github.com/confluentinc/schema-registry/blob/master/protobuf-serializer/src/main/java/io/confluent/kafka/serializers/protobuf/AbstractKafkaProtobufDeserializer.java#L139
  2. Get the message name from the schema https://github.com/confluentinc/schema-registry/blob/master/protobuf-serializer/src/main/java/io/confluent/kafka/serializers/protobuf/AbstractKafkaProtobufDeserializer.java#L140
  3. Make the FileDescriptorSet from the proto file got from schema registry (this is the blocker right now)
  4. Get the class using the message name
  5. Instantiate the class

Another alternative if using buf:
https://buf.build/docs/bsr/reflection/overview
https://googleapis.dev/python/protobuf/latest/google/protobuf/message_factory.html

@sauljabin sauljabin added the feature request New feature label Jul 16, 2024
@sauljabin sauljabin self-assigned this Jul 16, 2024
@sauljabin sauljabin changed the title Support descriptor pool for protobuf Support protobuf deserialization from Schema Registry Jul 16, 2024
@sauljabin sauljabin added this to the Kaskade v3 milestone Jul 16, 2024
@sauljabin sauljabin removed their assignment Jul 16, 2024
@sauljabin sauljabin changed the title Support protobuf deserialization from Schema Registry Support protobuf deserialization from Schema Registry and without a generated descriptor Jul 16, 2024
@sauljabin sauljabin added the blocked Depends on another issue label Aug 1, 2024
@sauljabin sauljabin removed this from the kaskade v3 milestone Nov 25, 2024
@sauljabin
Copy link
Owner Author

@sauljabin
Copy link
Owner Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocked Depends on another issue feature request New feature
Projects
None yet
Development

No branches or pull requests

1 participant