Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix primary scalability issue in distributor (camel) component #141

Closed
ThePrez opened this issue Jul 31, 2024 · 2 comments
Closed

Fix primary scalability issue in distributor (camel) component #141

ThePrez opened this issue Jul 31, 2024 · 2 comments

Comments

@ThePrez
Copy link
Owner

ThePrez commented Jul 31, 2024

Currently, the act of removing processed events from the database table (for instance, Manzan.ManzanMsg) imposes a substantial scalability roadblock.

The Camel route pulls off a set of events, processes them in parallel, and then each of those subtasks does a DELETE from before continuing with another iteration. The individual DELETE calls impedes a substantial overhead and the consumer cannot keep up if a lot of events are generated.

Some ideas:

  • Build custom code to handle the deletion from the table (for instance, have the Camel route add
  • Use JDBC batch processing for the multiple delete operations
  • Do the existing DELETE FROM asynchronously via the wiretap EIP
@ThePrez ThePrez added this to the Tech Preview 1 milestone Jul 31, 2024
@jonnyz32
Copy link
Collaborator

If we're just inserting into Manzan.ManzanMsg in order to read events from the table, why not just send messages via a socket instead? Or if we're worried about an extremely heavy load of messages, how about a streaming solution such as kafka?

@jonnyz32 jonnyz32 linked a pull request Sep 27, 2024 that will close this issue
@jonnyz32
Copy link
Collaborator

Fixed by #169 and #170

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants