Skip to content

Broker version compatibility

Jump to bottom
Magnus Edenhill edited this page May 10, 2016 · 13 revisions

librdkafka supports all released Apache Kafka broker versions since 0.8.0.0.0, but not all features may be available on all broker versions since some features rely on newer broker functionality.

Apache Kafka version 0.10.0.0 adds support for KIP-35 - querying the broker for supported API request types and versions - allowing the client to figure out what features it can use. But for older broker versions there is no way for the client to reliably know what the broker supports.

To alleviate this situation librdkafka adds three new configuration properties:

  • api.version.request=true|false - enables the API version request, this requires a >= 0.10.0.0 broker and will cause a disconnect on brokers 0.8.x - this disconnect is recognized by librdkafka and on the next connection attempt (which is immediate) it will disable the API version request and use broker.version.fallback as a basis of available features. NOTE: Due to a bug in broker version 0.9.0.0 & 0.9.0.1 the broker will not close the connection when receiving the API version request, instead the request will time out in librdkafka after 10 seconds and it will fall back to broker.version.fallback on the next immediate connection attempt.
  • broker.version.fallback=X.Y.Z.N - if the API version request fails (if api.version.request=true) or API version requests are disabled (api.version.request=false) then this tells librdkafka what version the broker is running and adapts its feature set accordingly.
  • api.version.fallback.ms=MS - In the case where api.version.request=true and the API version request fails, this property dictates for how long librdkafka will use broker.version.fallback instead of api.version.request=true. After MS has passed the API version request will be sent on any new connections made for the broker in question. This allows upgrading the Kafka broker to a new version with extended feature set without needing to restart or reconfigure the client (given that api.version.request=true).

Note: These properties applies per broker.

The API version query is currently disabled by default (api.version.request=false) in librdkafka (as of May 2016) due to the afforementioned bug in broker version 0.9.0.0 & 0.9.0.1.

NOTE: As more users move to broker version >= 0.10.0.0 the default value of api.version.request will be changed to true in librdkafka. Please read future release notes to know when this happens since it will affect users of <0.10.0 brokers.

Depending on what broker version you are using, please configure your librdkafka based client as follows:

Broker version >= 0.10.0.0 (or trunk)

api.version.request=true

Broker versions 0.9.0.x

api.version.request=false   # default, not need to set (yet)
broker.version.fallback=0.9.0.x  (the exact 0.9.0.. version you are using)

Broker versions 0.8.x.y

api.version.request=false   # default, not need to set (yet)
broker.version.fallback=0.8.x.y  (your exact 0.8... broker version)
Clone this wiki locally