Skip to content

Broker version compatibility

Jump to bottom
Magnus Edenhill edited this page Aug 19, 2017 · 13 revisions

librdkafka supports all released Apache Kafka broker versions since 0.8.0.0.0, but not all features may be available on all broker versions since some features rely on newer broker functionality.

Current default:

  • api.version.request=true (CHANGED to true in librdkafka v0.11.0)
  • broker.version.fallback=0.9.0.0

Depending on what broker version you are using, please configure your librdkafka based client as follows:

Broker version >= 0.10.0.0 (or trunk)

(this is the default value)

api.version.request=true

Broker versions 0.9.0.x

api.version.request=false
broker.version.fallback=0.9.0.x  (the exact 0.9.0.. version you are using)

Broker versions 0.8.x.y

api.version.request=false
broker.version.fallback=0.8.x.y  (your exact 0.8... broker version)

Detailed description

Apache Kafka version 0.10.0.0 adds support for KIP-35 - querying the broker for supported API request types and versions - allowing the client to figure out what features it can use. But for older broker versions there is no way for the client to reliably know what the broker supports.

To alleviate this situation librdkafka adds three new configuration properties:

  • api.version.request=true|false - enables the API version request, this requires a >= 0.10.0.0 broker and will cause a disconnect on brokers 0.8.x - this disconnect is recognized by librdkafka and on the next connection attempt (which is immediate) it will disable the API version request and use broker.version.fallback as a basis of available features. NOTE: Due to a bug in broker version 0.9.0.0 & 0.9.0.1 the broker will not close the connection when receiving the API version request, instead the request will time out in librdkafka after 10 seconds and it will fall back to broker.version.fallback on the next immediate connection attempt.
  • broker.version.fallback=X.Y.Z.N - if the API version request fails (if api.version.request=true) or API version requests are disabled (api.version.request=false) then this tells librdkafka what version the broker is running and adapts its feature set accordingly.
  • api.version.fallback.ms=MS - In the case where api.version.request=true and the API version request fails, this property dictates for how long librdkafka will use broker.version.fallback instead of api.version.request=true. After MS has passed the API version request will be sent on any new connections made for the broker in question. This allows upgrading the Kafka broker to a new version with extended feature set without needing to restart or reconfigure the client (given that api.version.request=true).

Note: These properties applies per broker.

The API version query was disabled by default (api.version.request=false) in librdkafka up to and including v0.9.5 due to the afforementioned bug in broker version 0.9.0.0 & 0.9.0.1, but was changed to true in librdkafka v0.11.0.

Clone this wiki locally