[improve][broker] Reduce unnecessary REPLICATED_SUBSCRIPTION_SNAPSHOT_REQUEST

[shibd]

Motivation

Let's say two clusters to two-way replication: cluster-1 and cluster-2

• cluster-1: has enabled subscription replication and correctly configured the cluster info for cluster-2
• cluster-2:has either incorrectly configured the information for cluster-1 or disabled subscription replication.

As a result, cluster-1 will send SNAPSHOT_REQUEST to cluster-2 at a frequency of replicatedSubscriptionsSnapshotFrequencyMillis (1 second by default).

In extreme cases, this can lead to a large number of consecutive marker between cluster-1 and cluster-2 topic.

This may affect the performance of reader and consumer, or cause reader.readNext() to time out.

Modifications

• When there is no complete snapshot, retry according to replicatedSubscriptionsSnapshotTimeoutSeconds to minimize the number of markers.

Verifying this change

• Add testReplicatedSubscriptionOneWay to cover this case.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 50.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 74.12%. Comparing base (bbc6224) to head (5f8ee19).
Report is 842 commits behind head on master.

Files with missing lines	Patch %	Lines
.../persistent/ReplicatedSubscriptionsController.java	50.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23839      +/-   ##
============================================
+ Coverage     73.57%   74.12%   +0.55%     
+ Complexity    32624    32187     -437     
============================================
  Files          1877     1853      -24     
  Lines        139502   143581    +4079     
  Branches      15299    16312    +1013     
============================================
+ Hits         102638   106430    +3792     
+ Misses        28908    28754     -154     
- Partials       7956     8397     +441     

Flag	Coverage Δ
inttests	26.62% <0.00%> (+2.04%)	⬆️
systests	23.11% <0.00%> (-1.21%)	⬇️
unittests	73.64% <50.00%> (+0.80%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
.../persistent/ReplicatedSubscriptionsController.java	75.65% <50.00%> (+3.06%)	⬆️

... and 1025 files with indirect coverage changes

[lhotari]

LGTM