OffsetForLeaderEpoch loop of failed requests with multiple leader changes

[emasab]

Description

Loop of OffsetForLeaderEpoch calls when multiple leader changes happen and one of them is being retried because of an error.

How to reproduce

To reproduce it there should be an initial partition leader change that triggers an OffsetForLeaderEpoch request. The request should fail while a second leader change happens. The corresponding current leader epoch isn't updated and there's a loop of failing requests.

Checklist

• librdkafka version (release number or git tag): 2.1.0+
• Apache Kafka version: 3.5.1
• librdkafka client configuration: any
• Operating system: any
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts
• Critical issue

[scanterog]

We've also hit this issue. Would it be feasible to have a 2.2.1 release as soon as the fix is ready?

[emasab]

@scanterog sure we're planning to include it in next release

[plachor]

Hi @emasab we also struggle with this one, once one of our instance is affected it consumes all CPU limit, and carry lots of RX (1GB/min) and TX transfer (100MB/min). Also broker is flooded with 400K+/min OffsetForLeaderEpoch requests.

We started to observe this recently once we decided to bump from v1.9.0 to v.2.2.0. Previously it was working fine. We observe this on maintenance actions:

• rolling brokers due to updates (OS and Kafka)
• scaling cluster - introducing new brokers and relocating partitions

I agree with @scanterog. Would it not be wise to cherry pick this and release HF of v2.2.0 and v2.1.0 for ones that are affected so they are not forced to pull minor release introducing new features as a HF to already deployed versions ?

edit: I'll just add that we use .net wrapper confluent-kafka-dotnet so this HF would need to be released there as well.

[rwkarg]

This continues to be an issue for us as well (using the same C# wrapper as @plachor. The OffsetForLeaderEpoch spam appears to be from inside librdkafka, not the .NET code).

Is there an estimated schedule for the next release that includes #4433?

The only resolution is to restart processes that get in this state and we haven't yet found a way to automate that as it's not in the .NET code where the issue is, and the monitoring we have on the broker side just says there's a log of this call getting made but not what client(s) it is coming from.

[ijuma]

This fix is included in 2.3.0 according to the release notes.