Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: support kafka schema registry for Protobuf and Avro. #579

Merged
merged 16 commits into from
Feb 26, 2024

Conversation

zliang-min
Copy link
Collaborator

PR checklist:

  • Did you run ClangFormat ?
  • Did you separate headers to a different section in existing community code base ?
  • Did you surround proton: starts/ends for new code in existing community code base ?

Please write user-readable short description of the changes:

New settings added for using kafka schema registry: kafka_schema_registry_url and kafka_schema_registry_credentials. Example:

CREATE EXTERNAL STREAM my_stream (
  -- columns goes here ...
) SETTINGS
type = 'kafka',
brokers = '...',
topic = '...',
data_format = 'ProtobufSingle',
kafka_schema_registry_url = 'http://url.to/my/schema/registry',
kafka_schema_registry_credentials = '$api_key}:${secret}';

kafka_schema_registry_credentials is optional, it contains the the API key and secret and a : in between. Also, kafka schema registry can be used with data_format = 'Avro'.

Note: in this PR, schema reference is not supported yet.

@chenziliang
Copy link
Collaborator

In general, it looks good and let's clean up the old code which is commented out

@chenziliang chenziliang reopened this Feb 26, 2024
@zliang-min zliang-min force-pushed the feature/protobuf-schema-registry branch from 90a0ae7 to aa4c34f Compare February 26, 2024 21:19
@zliang-min zliang-min merged commit c980957 into develop Feb 26, 2024
21 checks passed
@zliang-min zliang-min deleted the feature/protobuf-schema-registry branch February 26, 2024 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants