Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replicate closed indices #31141

Closed
2 of 8 tasks
dakrone opened this issue Jun 6, 2018 · 1 comment
Closed
2 of 8 tasks

Replicate closed indices #31141

dakrone opened this issue Jun 6, 2018 · 1 comment
Assignees
@tlrx
Labels
:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. >feature

Comments

@dakrone
Copy link
Member

dakrone commented Jun 6, 2018 •
edited
Loading

Closed indices currently have the downside that they are not replicated, so it's possible for an index to be closed, then the data lost because the node was terminated without saving the data.

We'd like to be able to close indices to remove any memory overhead of them, but still have the indices available for replication.

Steps to accomplish this (not necessarily in order):

There are a number of currently unresolved questions about this also:

  • Should we continue to allow restoring snapshots into closed indices? closed indices can't currently be snapshotted, so no breakage there
  • What should be done for indices restored from local store that contain an invalid index setting (we currently restore these as "closed")?
  • How should be handle settings that can only be changed when an index is closed and reopened, will they continue to work?
@dakrone dakrone added >feature :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. labels Jun 6, 2018
@dakrone dakrone self-assigned this Jun 6, 2018
@dakrone dakrone mentioned this issue Jun 6, 2018
Closed
8 tasks
dakrone added a commit to dakrone/elasticsearch that referenced this issue Jun 7, 2018
@dakrone
Add a NoopEngine implementation
c345e54 
This adds a new Engine implementation that does.. nothing. Any operations throw
an `UnsupportedOperationException` when tried. This engine is intended as a
building block for replicated closed indices in subsequent code changes.

Relates to elastic#31141
@dakrone dakrone mentioned this issue Jun 7, 2018
Merged
dakrone added a commit that referenced this issue Jun 15, 2018
@dakrone
Add a NoopEngine implementation (#31163)
e1d068d 
This adds a new Engine implementation that does.. nothing. Any operations throw
an `UnsupportedOperationException` when tried. This engine is intended as a
building block for replicated closed indices in subsequent code changes.

Relates to #31141
dakrone added a commit to dakrone/elasticsearch that referenced this issue Jun 18, 2018
@dakrone
Rename NoopEngine -> NoOpEngine
f86f71d 
In elastic#31163 (comment)
Jason requested renaming NoopEngine to NoOpEngine. This commit renames the class
and relevant parts.

Relates to elastic#31141
@dakrone dakrone mentioned this issue Jun 18, 2018
Merged
dakrone added a commit to dakrone/elasticsearch that referenced this issue Jun 18, 2018
@dakrone
Lazily retrieve EngineFactory based on IndexSettings
bd7a46f 
This commit changes the way that `EngineFactory`s are returned, instead of the
EngineFactory being set at time of shard creation, this changes the pluggability
so that an existing `IndexShard` can return a different engine factory depending
on dynamic index settings.

Relates to elastic#31141
@dakrone dakrone mentioned this issue Jun 18, 2018
Closed
dakrone added a commit that referenced this issue Jun 19, 2018
@dakrone
Rename NoopEngine -> NoOpEngine (#31398)
fc63e6c 
In #31163 (comment)
Jason requested renaming NoopEngine to NoOpEngine. This commit renames the class
and relevant parts.

Relates to #31141
dakrone added a commit to dakrone/elasticsearch that referenced this issue Jul 2, 2018
@dakrone
Use NoOp engine for replicating closed indices
513068f 
This commit uses the NoOp engine introduced in elastic#31163 for closed indices.
Instead of being removed from the routing table and essentially "dropped" (but
not deleted), closed indices are now replicated the same way open indices are.

Relates to elastic#31141
@dakrone dakrone mentioned this issue Jul 2, 2018
Closed
@dakrone dakrone assigned tlrx and unassigned dakrone Aug 7, 2018
tlrx pushed a commit that referenced this issue Sep 14, 2018
@dakrone @tlrx
Add a NoopEngine implementation (#31163)
1995804 
This adds a new Engine implementation that does.. nothing. Any operations throw
an `UnsupportedOperationException` when tried. This engine is intended as a
building block for replicated closed indices in subsequent code changes.

Relates to #31141
tlrx pushed a commit that referenced this issue Sep 14, 2018
@dakrone @tlrx
Rename NoopEngine -> NoOpEngine (#31398)
5741915 
In #31163 (comment)
Jason requested renaming NoopEngine to NoOpEngine. This commit renames the class
and relevant parts.

Relates to #31141
@tlrx tlrx mentioned this issue Sep 20, 2018
Closed
50 tasks
@tlrx
Copy link
Member

tlrx commented Sep 20, 2018

Closed in favor of #33888

@tlrx tlrx closed this as completed Sep 20, 2018
tlrx pushed a commit to tlrx/elasticsearch that referenced this issue Sep 26, 2018
@dakrone @tlrx
Add a NoopEngine implementation (elastic#31163)
04cb636 
This adds a new Engine implementation that does.. nothing. Any operations throw
an `UnsupportedOperationException` when tried. This engine is intended as a
building block for replicated closed indices in subsequent code changes.

Relates to elastic#31141
tlrx pushed a commit to tlrx/elasticsearch that referenced this issue Sep 26, 2018
@dakrone @tlrx
Rename NoopEngine -> NoOpEngine (elastic#31398)
580bef9 
In elastic#31163 (comment)
Jason requested renaming NoopEngine to NoOpEngine. This commit renames the class
and relevant parts.

Relates to elastic#31141
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tlrx tlrx

Labels
:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. >feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dakrone @tlrx