stats_cb and consumer group lag

[stripthesoul]

Is there a downfall of using the stats_cb to report consumer group lag for a topic? If we use the same consumer group from three applications, would the stats_cb topics/partitions lag be reported correctly across all three applications?

[stripthesoul]

@edenhill

[edenhill]

The stats consumer_lag is currently calculated as consumer_lag = hwmark - max(app_offset+1, committed_offset) where app_offset is the offset of the last message passed/returned to the application.

I think it should really be consumer_lag = hwmark - max(stored_offset, committed_offset) where stored_offset is the offset to be committed, basically app_offset +1 if enable.auto.offset.store=true(default) or the last manually stored offset (store_offsets()) if enable.auto.offset.store=false.
And I think we'll change this for the upcoming release since it would be more correct.

In either case, the stats consumer_lag is a cheaper and less error-prone way to monitor the consumer lag than to query the group coordinator for committed offsets.

[stripthesoul]

thanks thats reasonable, I do not auto commit, I leverage enable.auto.offset.store=false and then store offsets on my own. If two containers are running with the same consumer group, would the same results be correct if stats_cb is called for the same topic with the same consumer group but from two different threads?

[edenhill]

The consumer will only maintain consumer_lag for partitions that are assigned to it, but it should be up to date.