Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow to configure different implementations for Pulsar functions state store #12646

Merged
merged 1 commit into from
Nov 6, 2021
Merged

Allow to configure different implementations for Pulsar functions state store #12646

merged 1 commit into from
Nov 6, 2021

Conversation

merlimat
Copy link
Contributor

@merlimat merlimat commented Nov 5, 2021

Motivation

Added the option to configure different backend implementations for Pulsar Functions state store. Included implementation based on top of MetadataStore API.

This would be useful to avoid running BK TableService for standalone deployments.

@merlimat merlimat added type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages doc-not-needed Your PR changes do not impact docs labels Nov 5, 2021
@merlimat merlimat added this to the 2.10.0 milestone Nov 5, 2021
@merlimat merlimat self-assigned this Nov 5, 2021
eolivelli
eolivelli approved these changes Nov 6, 2021
View reviewed changes
Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like to make this pluggable.
I am not sure that writing to ZK is a good idea, probably it works for functions that do not read/write often the state

@merlimat
Copy link
Contributor Author

merlimat commented Nov 6, 2021

I like to make this pluggable.
I am not sure that writing to ZK is a good idea, probably it works for functions that do not read/write often the state

This is just one example of implementation and it’s using Metadata store interface, not ZK directly.

@merlimat merlimat merged commit 08a49c0 into apache:master Nov 6, 2021
zeo1995 pushed a commit to zeo1995/pulsar that referenced this pull request Nov 7, 2021
Merge branch 'up/master' into website/branch-2.7.2-chapter-5
c84b320 
* up/master: (55 commits)
  [broker] remove useless method "PersistentTopic#getPersistentTopic" (apache#12655)
  [Python Schema] Python schema support custom Avro configurations for Enum type (apache#12642)
  Allow to configure different implementations for Pulsar functions state store (apache#12646)
  Remove replicator global test from the quarantine group (apache#12648)
  [Java Client] Remove invalid call to Thread.currentThread().interrupt(); (apache#12652)
  k8s runtime: force deletion to avoid hung function worker during connector restart (apache#12504)
  [Broker] Optimize exception information for schemas (apache#12647)
  Close Zk database on unit tests (apache#12649)
  Fix call sync method in an async callback when enabling geo replicator. (apache#12590)
  [pulsar-broker] Add git branch information for PulsarVersion (apache#12541)
  PulsarAdmin: Fix last exit code storage (apache#12581)
  Add @test annotation to test methods (apache#12640)
  Upgrade debezium to 1.7.1 (apache#12644)
  [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry (apache#12606)
  [Functions] Prevent NPE while stopping a non started Pulsar LogAppender (apache#12643)
  Update io-debezium-source.md (apache#12638)
  Add missing cmds on pulsar-admin document page (apache#12634)
  Clean up the metadata of the non-persistent partitioned topics. (apache#12550)
  modify check waitingForPingResponse with volatile (apache#12615)
  [pulsar-admin] Check backlog quota policy for namespace (apache#12512)
  ...
zeo1995 pushed a commit to zeo1995/pulsar that referenced this pull request Nov 7, 2021
Merge branch 'up/master' into website/branch-2.7.2-chapter-6
497e20e 
* up/master: (55 commits)
  [broker] remove useless method "PersistentTopic#getPersistentTopic" (apache#12655)
  [Python Schema] Python schema support custom Avro configurations for Enum type (apache#12642)
  Allow to configure different implementations for Pulsar functions state store (apache#12646)
  Remove replicator global test from the quarantine group (apache#12648)
  [Java Client] Remove invalid call to Thread.currentThread().interrupt(); (apache#12652)
  k8s runtime: force deletion to avoid hung function worker during connector restart (apache#12504)
  [Broker] Optimize exception information for schemas (apache#12647)
  Close Zk database on unit tests (apache#12649)
  Fix call sync method in an async callback when enabling geo replicator. (apache#12590)
  [pulsar-broker] Add git branch information for PulsarVersion (apache#12541)
  PulsarAdmin: Fix last exit code storage (apache#12581)
  Add @test annotation to test methods (apache#12640)
  Upgrade debezium to 1.7.1 (apache#12644)
  [ML] Avoid passing OpAddEntry across a thread boundary in asyncAddEntry (apache#12606)
  [Functions] Prevent NPE while stopping a non started Pulsar LogAppender (apache#12643)
  Update io-debezium-source.md (apache#12638)
  Add missing cmds on pulsar-admin document page (apache#12634)
  Clean up the metadata of the non-persistent partitioned topics. (apache#12550)
  modify check waitingForPingResponse with volatile (apache#12615)
  [pulsar-admin] Check backlog quota policy for namespace (apache#12512)
  ...
eolivelli pushed a commit that referenced this pull request Nov 9, 2021
@merlimat @eolivelli
Allow to configure different implementations for Pulsar functions sta…
5fdb94e 
…te store (#12646)

(cherry picked from commit 08a49c0)
eolivelli pushed a commit to eolivelli/pulsar that referenced this pull request Nov 29, 2021
@merlimat @eolivelli
Allow to configure different implementations for Pulsar functions sta…
6912946 
…te store (apache#12646)
@merlimat merlimat deleted the state-store-metadata-impl branch November 29, 2021 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eolivelli eolivelli eolivelli approved these changes

@sijie sijie Awaiting requested review from sijie

@freeznet freeznet Awaiting requested review from freeznet

@jerrypeng jerrypeng Awaiting requested review from jerrypeng

@hangc0276 hangc0276 Awaiting requested review from hangc0276

@codelipenghui codelipenghui Awaiting requested review from codelipenghui

Assignees

@merlimat merlimat

Labels
doc-not-needed Your PR changes do not impact docs type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages
Projects
None yet
Milestone
2.10.0
Development

Successfully merging this pull request may close these issues.
2 participants
@merlimat @eolivelli