Making partitionConsumer claim loop more robust

[rengawm]

This attempts to resolve two race conditions we've noticed while using this library:

1. Timeout race condition: Consumer A is handling partition 1. It has recently grabbed a batch of messages from Kafka on that partition to process. Consumer B comes online, triggering a rebalance of partition 1 from consumer A to B. However, consumer A is taking a while to finish the batch it last received, so it eventually times out after config.Offsets.ProcessingTimeout. However, consumer B is in a retry loop also taking a maximum of config.Offsets.ProcessingTimeout. If it fails to get the partition in that amount of time, it will give up. Since both are working off of the same timeout value, this is sufficient to sometimes cause consumer B to fail claiming the partition even though consumer A has just released it. To resolve this, I've added two more retry iterations to ensure that we will perform a couple more attempts even after this timeout value.

2. The root issue is actually in kazoo-go, in https://github.com/wvanbergen/kazoo-go/blob/master/consumergroup.go#L273 but I've updated the claim retry loop here to handle this and any other error cases. But for background, here's what we observed:

• Consumer A owns partition 1
• Consumer B comes online, triggering rebalance of partition 1 from A to B
• Consumer B attempts to create zookeeper node at /consumers/<cgName>/owners/<topicName>/<partitionNumber>, but fails since it already exists
• Consumer A now deletes /consumers/<cgName>/owners/<topicName>/<partitionNumber>
• Consumer B attempts to determine who owns the partition by reading /consumers/<cgName>/owners/<topicName>/<partitionNumber>, but it's now gone so that fails.

That specific race condition may also require a more targeted fix, but more broadly, we think that if a consumer is trying to claim a partition, it should continue trying to claim it rather than ending up in a state where the partition is unowned indefinitely. As such, the retry loop here will continue running regardless of the error we get until we hit the retry limit or successfully claim the partition. Also, if we ultimately fail to claim it, we report the error to the consumergroup's error channel so the client can determine how to handle the failure to claim a partition.

[mihasya]

@wvanbergen @eapache any chance this can get some attention? We are running all our service with this patch and it has completely eliminated this particular error case for us.

[NeoCN]

@wvanbergen @eapache we have run into the same issue too! any update on this?