Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Create system indices with a 1 primary, 0-all replica sharding strategy #846

Open
engechas opened this issue Feb 8, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@engechas
Copy link
Collaborator

engechas commented Feb 8, 2024

Is your feature request related to a problem?
Some of the existing system indices necessary for security analytics work are created with a 5p/1r shard strategy. The log types config is created with a 5p/0-allr strategy. Both of these have issues:

  1. 5p/1r - depending on the data node count, this can lead to shard skew. For some of the more heavily queried indices, such as the detector queries index, this can lead to hot nodes.
  2. 5p/0-allr - for large clusters, this results in a very high shard count. Taking a 20 node cluster as an example, this index would create 100 shards

What solution would you like?

  1. System indices should use a 1p/0-allr strategy. This allows each node to hold a copy of the entire index's data so any queries against these indices can be executed on the local node, improving performance and avoiding skew. Rollover should continue to be used for indices that can grow larger than a single shard should be.
@engechas engechas added enhancement New feature or request untriaged labels Feb 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Backlog (Feature Requests, Enhancements)
Development

No branches or pull requests

2 participants