Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a new Engine implementation for replicas with segment replication enabled. #3240

Merged
merged 16 commits into from
May 24, 2022
Merged

Add a new Engine implementation for replicas with segment replication enabled. #3240

merged 16 commits into from
May 24, 2022

Commits on May 6, 2022

  1. Change fastForwardProcessedSeqNo method in LocalCheckpointTracker to … 

    …persisted checkpoint.

This change inverts fastForwardProcessedSeqNo to fastForwardPersistedSeqNo for use in
Segment Replication.  This is so that a Segrep Engine can match the logic of InternalEngine
where the seqNo is incremented with each operation, but only persisted in the tracker on a flush.
With Segment Replication we bump the processed number with each operation received index/delete/noOp, and
invoke this method when we receive a new set of segments to bump the persisted seqNo.

Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 6, 2022
    Configuration menu
    Copy the full SHA
    8553d2f View commit details
    Browse the repository at this point in the history

Commits on May 9, 2022

  1. Extract Translog specific engine methods into an abstract class. 

    This change extracts translog specific methods to an abstract engine class so that other engine
implementations can reuse translog logic.

Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 9, 2022
    Configuration menu
    Copy the full SHA
    5c54323 View commit details
    Browse the repository at this point in the history

  2. Add a separate Engine implementation for replicas with segment replic… 

    …ation enabled.

This change adds a new engine intended to be used on replicas with segment replication enabled.
This engine does not wire up an IndexWriter, but still writes all operations to a translog.
The engine uses a new ReaderManager that refreshes from an externally provided SegmentInfos.

Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 9, 2022
    Configuration menu
    Copy the full SHA
    e77b616 View commit details
    Browse the repository at this point in the history

  3. Fix spotless checks. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 9, 2022
    Configuration menu
    Copy the full SHA
    22e7dde View commit details
    Browse the repository at this point in the history

  4. Fix :server:compileInternalClusterTestJava compilation. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 9, 2022
    Configuration menu
    Copy the full SHA
    daacef4 View commit details
    Browse the repository at this point in the history

  5. Fix failing test naming convention check. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 9, 2022
    Configuration menu
    Copy the full SHA
    3d3cf48 View commit details
    Browse the repository at this point in the history

Commits on May 11, 2022

  1. PR feedback. 

    - Removed isReadOnlyReplica from overloaded constructor and added feature flag checks.
- Updated log msg in NRTReplicationReaderManager
- cleaned up store ref counting in NRTReplicationEngine.

Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 11, 2022
    Configuration menu
    Copy the full SHA
    8fddec8 View commit details
    Browse the repository at this point in the history

  2. Fix spotless check. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 11, 2022
    Configuration menu
    Copy the full SHA
    576f61d View commit details
    Browse the repository at this point in the history

Commits on May 12, 2022

  1. Remove TranslogAwareEngine and build translog in NRTReplicationEngine. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 12, 2022
    Configuration menu
    Copy the full SHA
    e0d9742 View commit details
    Browse the repository at this point in the history

  2. Fix formatting 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 12, 2022
    Configuration menu
    Copy the full SHA
    41715d5 View commit details
    Browse the repository at this point in the history

  3. Add missing translog methods to NRTEngine. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 12, 2022
    Configuration menu
    Copy the full SHA
    4201475 View commit details
    Browse the repository at this point in the history

  4. Remove persistent seqNo check from fastForwardProcessedSeqNo. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 12, 2022
    Configuration menu
    Copy the full SHA
    675dc11 View commit details
    Browse the repository at this point in the history

  5. PR feedback. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 12, 2022
    Configuration menu
    Copy the full SHA
    5ea0e7b View commit details
    Browse the repository at this point in the history

  6. Add test specific to translog trimming. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 12, 2022
    Configuration menu
    Copy the full SHA
    5bccdd8 View commit details
    Browse the repository at this point in the history

  7. Javadoc check. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 12, 2022
    Configuration menu
    Copy the full SHA
    5b380d2 View commit details
    Browse the repository at this point in the history

Commits on May 18, 2022

  1. Add failEngine calls to translog methods in NRTReplicationEngine. 

    Roll xlog generation on replica when a new commit point is received.

Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed May 18, 2022
    Configuration menu
    Copy the full SHA
    06b0d98 View commit details
    Browse the repository at this point in the history