How to improve resilience to Kafka maintenance

[abtreece]

Every time our Kafka service undergoes maintenance one of our producer applications is incapable of handling the broker shuffling. Messages back up andbrod seemingly does not reconnect to Kafka and process the backed up messages.

We begin seeing these kinds of errors logged for messages...

[error] Produce error whitelist-19 Offset: -1 Error: :not_leader_for_partition
[error] Produce error whitelist-31 Offset: -1 Error: :not_enough_replicas
[error] Produce error whitelist-8 Offset: 419723027 Error: :not_enough_replicas_after_append

and then...

2021-11-08 02:11:11.748 [error] GenServer #PID<0.13570.0> terminating
** (stop) {:reached_max_retries, {:produce_response_error, "whitelist", 19, -1, :not_enough_replicas}}
    (brod) /opt/build/deps/brod/src/brod_producer_buffer.erl:304: :brod_producer_buffer.rebuffer_or_crash/3
    (brod) /opt/build/deps/brod/src/brod_producer.erl:390: :brod_producer.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:msg, #PID<0.13584.0>, {:kpro_rsp, #Reference<0.391829740.1032060931.101778>, :produce, 5, %{responses: [%{partition_responses: [%{base_offset: -1, error_code: :not_enough_replicas, log_append_time: -1, log_start_offset: 385207009, partition: 19}], topic: "whitelist"}], throttle_time_ms: 0}}}

2021-11-08 02:11:11.748 [error] Process #PID<0.13570.0> terminating
** (exit) {:reached_max_retries, {:produce_response_error, "whitelist", 19, -1, :not_enough_replicas}}
    (brod) /opt/build/deps/brod/src/brod_producer_buffer.erl:304: :brod_producer_buffer.rebuffer_or_crash/3
    (brod) /opt/build/deps/brod/src/brod_producer.erl:390: :brod_producer.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Initial Call: :brod_producer.init/1
Ancestors: [#PID<0.13550.0>, #PID<0.905.0>, :brod_client_1, :brod_sup, #PID<0.869.0>]

2021-11-08 02:11:11.748 [error] Child 19 of Supervisor #PID<0.13550.0> (:brod_producers_sup) terminated
** (exit) {:reached_max_retries, {:produce_response_error, "whitelist", 19, -1, :not_enough_replicas}}
Pid: #PID<0.13570.0>
Start Call: :brod_producer.start_link(#PID<0.878.0>, "whitelist", 19, [])

our brod_config...

  def brod_config do
    [
      auto_start_producers: true,
      ssl: ssl_config(),
      config: [
        restart_delay_seconds: 10,
        max_metadata_sock_retry: 1,
        get_metadata_timeout_seconds: 5,
        reconnect_cool_down_seconds: 1,
        allow_topic_auto_creation: false,
        default_producer_config: [
          # require ACK from all Kafka in-sync replicas
          required_acks: -1
        ]
      ]
    ]
  end

Are there changes that we can/should make to our brod_config to improve resilience?

[zmstone]

Hi @abtreece you can try to increase the number of allowed retries.
max_retries, and add more delay between retries retry_backoff_ms

or use this producer instead:
https://github.com/kafka4beam/wolff#more-resilient-to-network-and-kafka-disturbances