Remove Shadow Replicas

[bleskes]

The shadow replicas feature (added in #8976) has not seen the amount of usage we originally hoped for. On the flip side it is very tricky to maintain, effectively adding a second replication logic. Example issues include #22021 , #17695, #16358 , #13583 , #16357 - see here for more. It is also requires new features (like sequence numbers) to be implemented for both normal replicas and shadow replicas, increasing the amount of future work.

Given the lack of uptake we are considering to remove them (and will mark them as deprecated in a separate PR).

This is issue is opened as a placeholder for discussion and to give people the option to object and state their use case.

[makeyang]

segment-based or segments-based replication is really helpful for certain scenario.let's say to provide something like read-write isloate which is common in database.
but this feature shouldn't bound to shared filesystem.

[bleskes]

@makeyang yes. Segment based replication represent different trade offs and is interesting in some aspects. As you say - it is not bound to a shared file system. We talking (for a long time) about the cons and pros of it and how to potentially implement it without relying on a shared file system to do the heavy lifting of transferring files between nodes (note that we currently rely on the translog to be copied as well, something segment based replication doesn't imply). If and when we implement it, it should work on normal file systems. Since that's the hardest part to address, I don't feel there is value in keeping shadow replicas to expedite segment based replication (if this is what you meant).

[makeyang]

@bleskes got it. thanks. if translog based or segment based asynchronous replication is implemente, it is really no reaso to keep shadow replicas.

[chenryn]

Is this meaning asynchronous replica will be published when shadow replica is totally removed?

[jasontedor]

Is this meaning asynchronous replica will be published when shadow replica is totally removed?

It does not mean that.

[apatrida]

what about read heavy scenarios where you want to scale uniformly per node without the delay of replication (segment or per transaction) or without the cost of applying them (per transaction) and maybe over a WAN (segment would work for that, since likely it is async)?

[apatrida]

Other question related to the future of replacing this, because not sure where thinking is represented (other github issue) for future replication models.

WAN replication => the clusters are not one cluster, but two distinct and may have different shard balances. Poor man's version of this now is near-cluster=>S3 backup/snapshot=>S3 restore snapshot=>remote-cluster

[bleskes]

what about read heavy scenarios where you want to scale uniformly per node without the delay of replication (segment or per transaction) or without the cost of applying them (per transaction)

I'm not sure what you mean? a ready heavy environment means writing is not a big deal, and most of the work of the shards is to serve reads. Shadow replicas, which still need to push the segments (with the few new documents) and do stuff like merging, won't help much there?

maybe over a WAN
Poor man's version of this now is near-cluster=>S3 backup/snapshot=>S3 restore snapshot=>remote-cluster

I need, snapshot and restore is the current solution. Shadow replicas are not a long term solution as the current implementation assumes a shared file system. We are working on stuff like changes API which will allow you to stream (with delay) operations to another cluster.

[jrj0823]

Have you considered running ES directly on a shared filesystem?
I notice that after shadow replica is removed, there is no feature to write data directly to shared filesystem.
I think read ability can be scaled more quickly and at low cost with a shared filesystem.
If shared filesystem supports tiered storage like EC, we do not need to handle the migration of hot and cold data.

[DaveCTurner]

Have you considered running ES directly on a shared filesystem?

Yes, see this blog post for news about developments in that direction.