Skip to content
This repository has been archived by the owner on Dec 13, 2023. It is now read-only.

AMQP enhancements for resiliency and concurrency #2587

Merged
merged 31 commits into from
Feb 23, 2022
Merged

AMQP enhancements for resiliency and concurrency #2587

merged 31 commits into from
Feb 23, 2022

Conversation

venkathangella
Copy link
Contributor

@venkathangella venkathangella commented Nov 22, 2021 •
edited
Loading

Pull Request type

  • Feature

Changes in this PR

  • Even based message processing
  • Multi-channel support for concurrent message publishing
  • Connection and channel resiliency improvements
    Retries and validity checks
  • Publish Retries
  • Ack retries
  • Channel reusability
  • Extended the configuration for resiliency and availability
  • Quorum Queue Support
  • Redis connection validity checks

Describe the new behavior from this PR, and why it's needed
Issue #

  1. Improves the AMQP based workflow processing capabilities of the conductor as messages get processed for each message instead of a scheduled collection of messages.
  2. Quorum queue support enhances data safety.
  3. Messages get published concurrently instead of waiting for a single channel.
  4. Heartbeats to keep the connections active.
  5. Code changes to keep the Redis connection active in a connection pool.

Alternatives considered

Describe alternative implementation you have considered

Thangella, Venkat and others added 19 commits October 28, 2021 10:43
AMQP messages will generate the observable change
c76d9fd
Reverting unnecessary changes
d752e85
Capturing the only changes required for AMQP messagres reading
4233d39
Capturing the only changes required for AMQP messagres reading
44481b5
Support for sequential and parallel processing of messages
c054c7c
Merge branch 'main' of https://github.com/venkathangella/conductor in…
31cc9a7 
…to conductor_changes
@venkathangella
Merge branch 'Netflix:main' into conductor_changes
0462495
Channel reusability changes
7859216
Channel reusability changes
7caff8a
Subscriber channel caching
d8dbeef
Max channel changes
d4a4910
Connection and opertion retries
13d67bf
Channel usage for exchanges and publishers
1d73545
Acknowledge the message even on failures
d5d6e03
Resolved conflicts
5f60f3b
Resolved conflicts
eb52139
Apply coding style changes using spotless
f0574cb
Merge branch 'main' of https://github.com/venkathangella/conductor in…
4700ea3 
…to conductor_changes
Retry on publish and ack failures
709e7e4
@venkathangella venkathangella changed the title Conductor changes AMQP enhancements for resiliency and concurrency Jan 20, 2022
Thangella, Venkat and others added 10 commits January 21, 2022 05:14
Externalizing the redis config
d51c8f9
Externalizing the redis config
8fb9244
@v1r3n @venkathangella
Make Event System Task Synchronous
1ddb5c8 
Event is a task that publishes messages to an external eventing system.
The publishing is not a long running task, and publishing message to an async system using async task is not optimal.

marking event as sync, improves the overall performance and has negligible overhead given the task completes quite quick.
@venkathangella
Merge branch 'Netflix:main' into conductor_changes
df89bc2
Merge branch 'main' of https://github.com/venkathangella/conductor in…
d096e5d 
…to conductor_changes
Merge branch 'conductor_changes' of https://github.com/venkathangella…
f6c4c77 
…/conductor into conductor_changes
@v1r3n @venkathangella
Update integration tests to reflect Event is synchronous
f14fce1
Merge branch 'conductor_changes' of https://github.com/venkathangella…
509927d 
…/conductor into conductor_changes
Getting the latest changes
09f8645
Merge branch 'main' of https://github.com/venkathangella/conductor in…
c6be1f3 
…to conductor_changes
@venkathangella venkathangella marked this pull request as ready for review February 11, 2022 04:46
@apanicker-nflx
Copy link
Collaborator

@ritu-p @guru1306 Please help with reviewing this PR to the AMQP event queue module

guru1306
guru1306 reviewed Feb 17, 2022
View reviewed changes
contribs/src/main/java/com/netflix/conductor/contribs/queue/amqp/AMQPConnection.java Show resolved Hide resolved
contribs/src/main/java/com/netflix/conductor/contribs/queue/amqp/AMQPConnection.java Show resolved Hide resolved
contribs/src/main/java/com/netflix/conductor/contribs/queue/amqp/AMQPObservableQueue.java Show resolved Hide resolved
contribs/src/main/java/com/netflix/conductor/contribs/queue/amqp/AMQPObservableQueue.java Show resolved Hide resolved
contribs/src/main/java/com/netflix/conductor/contribs/queue/amqp/AMQPObservableQueue.java Show resolved Hide resolved
...src/main/java/com/netflix/conductor/contribs/queue/amqp/config/AMQPEventQueueProperties.java Show resolved Hide resolved
contribs/src/main/java/com/netflix/conductor/contribs/queue/amqp/config/AMQPRetryPattern.java Show resolved Hide resolved
contribs/src/test/java/com/netflix/conductor/contribs/queue/amqp/AMQPObservableQueueTest.java Show resolved Hide resolved
core/src/main/java/com/netflix/conductor/core/execution/tasks/Event.java Show resolved Hide resolved
dependencies.gradle Show resolved Hide resolved
@venkathangella
Copy link
Contributor Author

@apanicker-nflx , could you please merge this as we have got the required approvals?

@apanicker-nflx apanicker-nflx merged commit 9ffcabc into Netflix:main Feb 23, 2022
v1r3n added a commit to Netflix/conductor-community that referenced this pull request Feb 25, 2022
@v1r3n
AMQP Netflix/conductor#2587 merge to communnity
00607b5 
merge changes from Netflix/conductor#2587 into community repo
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@ritu-p ritu-p ritu-p approved these changes

@guru1306 guru1306 guru1306 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@venkathangella @apanicker-nflx @ritu-p @guru1306 @v1r3n