Home

Welcome to the azure-functions-kafka-extension wiki!

Q&A

Q1. If I scale out a function app with an Elastic Premium Plan and run it on multiple instances, does each instance belong to the same Consumer Group?

A consumer group is related to a function not to an instance. For example, If the app contains only one function that belongs to one consumer group, then in scaling out - each instance will belong to the same consumer group.

java sample

@KafkaTrigger(topic = "",
        brokerList="",
        consumerGroup="",
        protocol = BrokerProtocol.SASLSSL,
        authenticationMode = BrokerAuthenticationMode.PLAIN,
        username = "",
        password = ""
)

Q2. Where is Kafka's offset stored?

It is on the Kafka server. Each topic contains the offset for each partition. The function infrastructure does not store the offset but retrieves it on each invocation.

Q3. Do multiple instances share offsets in time and read the same message?

Given that the same consumer group is used, messages can be processed more than once, either by Kafka or due to a rebalance of partitions (function instance changes). It is highly recommended to handle at-least once message delivery. Even though messages might be processed more than once, they will be always in order.

Q4. If the number of instances (consumers) is greater than the number of Kafka partitions, will there be a consumer that is not involved in the operation?

Generally speaking Number of instances cannot exceed the number of partitions. But if this happens the extra consumer will not be involved.

Q5 How does scale out/in happen in the Elastic Premium Plan?

If the messages is processed fast, then the system will scale in as there is no increase in the messages rate and messages does not increase 1000 unprocessed message. If the messages is processed slowly, then the system will scale out to the number of partitions per topic per consumer group.