Proposal: log progress of recovery

[TheL1ne]

When we create new instances which have to sync from scratch with kafka we have no clue if the instance is still alive while it recovers the data from kafka which can lead to uncertainty over multiple hours.
It would be great to log at least how much was already synced and it would be even better when we can log an approx of how long it will take to recover the remaining missing data.

[db7]

Predicting won't work. There are two problems:
1- The only thing you know is the current offset and the high-water mark. You don't know how many of the records have been deleted because they may be updates to the same keys. So (HWM-offset) don't tell you how many records you still really have to read from a compacted topic.
2- Even if you would use (HWM-offset) as an upper bound, you cannot know how long LevelDB will need to do its compaction after the state is restored -- and in some cases that takes longer than reading the records from Kafka.

Stats() already gives you information of how many bytes are being read from which topic. You can use that as a rule-of-thumb for the specific application to guess how long it will take to recover. Only when the LevelDB compaction is running that you won't see any Kafka traffic for the topic/partition being recovered. During that period one should probably look at disk IO and CPU to guess if there is progress.

[TheL1ne]

If we cannot approx the future I would at least add some logging to show that we are still alive

[db7]

I guess this has been merged now, right? Can I close the ticket?

[frairon]

You're right, let's close it