Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CCR] Limit the write buffer by memory rather than op count #34705

Closed
bleskes opened this issue Oct 22, 2018 · 3 comments
Closed

[CCR] Limit the write buffer by memory rather than op count #34705

bleskes opened this issue Oct 22, 2018 · 3 comments
Assignees
Labels
:Distributed Indexing/CCR Issues around the Cross Cluster State Replication features

Comments

@bleskes
Copy link
Contributor

bleskes commented Oct 22, 2018

Currently we stop sending read requests if the number of operation waiting to sent as a write bulk is higher than a given number. That was the simplest to implement but it makes it harder to reason about a good value - the ultimate goal of setting a limit is to protect against memory consumption. I think we should move to a byte sized based limit.

/cc @jasontedor @martijnvg

@bleskes bleskes added :Distributed Indexing/CCR Issues around the Cross Cluster State Replication features team-discuss labels Oct 22, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@jasontedor
Copy link
Member

This was part of the initial plan, we should implement this now indeed. I discussed this with @martijnvg in another channel and he will proceed with the implementation.

One thing I'm unsure of is this should be a move to a byte-sized based limit as opposed to having the possibility of either (and enforcing both). I lean towards the latter.

@bleskes
Copy link
Contributor Author

bleskes commented Oct 23, 2018

I'm fine with having both but I think a bytes (i.e. memory) based one is enough. I think it's the main motivation to limit it. I struggle to find a good default for the count based one, but it's easy to have sane default (1GB or 512MB) for the bytes.

martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Oct 24, 2018
This limit is based on the size in bytes of the operations in the write buffer. If this limit is exceeded then no more read operations will be coordinated until the size in bytes of the write buffer has dropped below the configured write buffer size limit.

Renamed existing `max_write_buffer_size` to ``max_write_buffer_count` to indicate that limit is count based.

Closes elastic#34705
martijnvg added a commit that referenced this issue Oct 24, 2018
This limit is based on the size in bytes of the operations in the write buffer. If this limit is exceeded then no more read operations will be coordinated until the size in bytes of the write buffer has dropped below the configured write buffer size limit.

Renamed existing `max_write_buffer_size` to ``max_write_buffer_count` to indicate that limit is count based.

Closes #34705
martijnvg added a commit that referenced this issue Oct 24, 2018
This limit is based on the size in bytes of the operations in the write buffer. If this limit is exceeded then no more read operations will be coordinated until the size in bytes of the write buffer has dropped below the configured write buffer size limit.

Renamed existing `max_write_buffer_size` to ``max_write_buffer_count` to indicate that limit is count based.

Closes #34705
kcm pushed a commit that referenced this issue Oct 30, 2018
This limit is based on the size in bytes of the operations in the write buffer. If this limit is exceeded then no more read operations will be coordinated until the size in bytes of the write buffer has dropped below the configured write buffer size limit.

Renamed existing `max_write_buffer_size` to ``max_write_buffer_count` to indicate that limit is count based.

Closes #34705
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed Indexing/CCR Issues around the Cross Cluster State Replication features
Projects
None yet
Development

No branches or pull requests

4 participants