Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redis pubsub improvement #339

Merged
merged 9 commits into from
Jul 19, 2024

Conversation

RemindD
Copy link
Contributor

@RemindD RemindD commented Jul 9, 2024

  • Subscriber launch two async threads to
    • reclaim the ownership on messages that are previously claimed by its client (scan once per 5 seconds, claim again if the message not claimed for 30s)
    • claim the messages that are previously claimed by other replica clients, but the previous client has been dead for a while (scan once per 1 minutes, claim again if the message not claimed for 60s)
  • Added an in-memory set to track what messages are being processed by handler
    • add the message id to the set before handler starts processing the message and remove the message id when handler either succeeds or fails
    • Every time a message is claimed by subscriber async thread, the provider will decide whether the handler should be invoked for this message depending on whether the message id is in the set
  • bump redis client from v7 to v9
  • support symphony multiple replica scenario

@RemindD RemindD marked this pull request as draft July 9, 2024 06:54
@RemindD RemindD marked this pull request as ready for review July 9, 2024 13:53
Haishi2016
Haishi2016 previously approved these changes Jul 15, 2024
Copy link
Contributor

@Haishi2016 Haishi2016 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great PR.

@msftcoderdjw
Copy link
Contributor

Do we have multiple subscribers to subscribe to same topic but take different actions? This is not the same as same subscribers with multiple instances. The former case should let different subscriber consume every message. The latter case is once one subscriber execution routine claim the message, the other routine doesn't need to take that message.

@RemindD
Copy link
Contributor Author

RemindD commented Jul 16, 2024

Do we have multiple subscribers to subscribe to same topic but take different actions? This is not the same as same subscribers with multiple instances. The former case should let different subscriber consume every message. The latter case is once one subscriber execution routine claim the message, the other routine doesn't need to take that message.

I have some ideas about this and will work out another PR for single publisher/multiple subscriber mode.

@msftcoderdjw msftcoderdjw merged commit 639d7c2 into eclipse-symphony:main Jul 19, 2024
5 checks passed
@msftcoderdjw msftcoderdjw deleted the users/xingdong/pubsub branch July 19, 2024 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants