Create committableSink as combination of Producer.flexiFlow an…
#932
Conversation
|
As an alternative to this, the combination could be pre-packaged for now and get combined implementation with a specific stage later. |
There was a problem hiding this comment.
As an alternative to this, the combination could be pre-packaged for now and get combined implementation with a specific stage later.
I think this is the best path forward. Also, I noticed there are some deprecated Producer.committerSink methods from 1.0-RC1. Are these safe to remove in 2.0?
ennru
changed the title
committingSink as combination of Producer.flexiFlow and Committer.sink
|
Re-targeted this PR to introduce new sinks for the produce-and-commit combination. The existing |
| @@ -152,6 +153,116 @@ object Producer { | |||
| producer: org.apache.kafka.clients.producer.Producer[K, V] | |||
| ): Sink[Envelope[K, V, ConsumerMessage.Committable], CompletionStage[Done]] = committableSink(settings, producer) | |||
|
|
|||
| /** | |||
| * Create a sink that is aware of the [[ConsumerMessage.CommittableOffset committable offset]] | |||
There was a problem hiding this comment.
Typo [[ConsumerMessage.CommittableOffset committable offset]]. The other new factory methods in the Scala and Java DSLs have the same typo.
There was a problem hiding this comment.
It's not a typo a link to a class or method can use any text, but I figured these links should link to Committable instead.
There was a problem hiding this comment.
Got it. I misunderstood the scaladocs syntax.
| * Note that there is a risk that something fails after publishing but before | ||
| * committing, so it is "at-least once delivery" semantics. | ||
| */ | ||
| def sinkWithOffsetContext[K, V, IN <: Envelope[K, V, _], C <: CommittableOffset]( |
There was a problem hiding this comment.
Should it be called committingSinkWithOffsetContext? I know it's a mouthful..
There was a problem hiding this comment.
I changed to sinkWithCommitting and sinkWithCommittingOffsetContext instead.
|
I had to upgrade to Scala 2.13.1 as scaladoc hit some bug. |
ennru
changed the title
committingSink as combination of Producer.flexiFlow and Committer.sinksinkWithCommitting as combination of Producer.flexiFlow and Committer.sink
ennru
force-pushed
the
produce-and-commit-docs
branch
from
a4cfdd4 to
9295c3e
Compare
ennru
changed the title
sinkWithCommitting as combination of Producer.flexiFlow and Committer.sinkcommittableSink as combination of Producer.flexiFlow and Committer.sink
|
Ok, @seglo and I figured |
ennru
changed the title
committableSink as combination of Producer.flexiFlow and Committer.sinkcommittableSink as combination of Producer.flexiFlow an…
Purpose
For the common consume-produce flow, a combination of producer and committer is useful. It conceptually consists of a
Producer.flexiFlowwith aCommitter.sink, but could be implemented as a specific sink stage later.Changes
committableSinkwhich combinesflexiFlowwithCommitter.sinkcommittableSinkWithOffsetContextwhich combinesflowWithOffsetContextwithCommitter.sinkWithOffsetContext(but is implemented simpler than that)Background Context
The committer uses commit batching internally while the producer
committableSinksends commits one-by-one.