[Remote Store + Segrep] Support parallel segment file upload & download in replication path

[ashking94]

Is your feature request related to a problem? Please describe.
Currently the segments upload from primary and segments download on replicas is sequential. Only after the segments from primary are uploaded and subsequently the segment metadata file uploaded, the primary publishes to the replicas about initiating the segments sync. Given we have parallel segment files upload, we can initiate downloads on replicas after each file upload. This way we can reduce the overall time for replication.

Describe the solution you'd like
The segrep publisher can publish to replicas side along with individual segment files getting uploaded. We can decide the exact approach based on discussion.

Describe alternatives you've considered
NA

Additional context
NA

[ashking94]

@mch2 @sachinpkale @Bukhtawar @gbbafna @ankitkala What do you think?

[anasalkouz]

@kotwanikunal is this overlapped with multipart download?

[kotwanikunal]

@kotwanikunal is this overlapped with multipart download?

This seems more like an enhancement if I understand correctly. The approach defined here will start the segment downloads on replicas as soon as the upload is complete on primary, instead of waiting for all the segments to be uploaded. Is that that the correct understanding @ashking94 ?

[mch2]

I think this is worth exploring for sure but isn't a trivial refactor. We could maybe publish checkpoints with expected files up front and then poll the store on an interval/backoff. Or we publish as files become available.