-
Notifications
You must be signed in to change notification settings - Fork 25k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CCR] Limit the write buffer by memory rather than op count #34705
Comments
Pinging @elastic/es-distributed |
This was part of the initial plan, we should implement this now indeed. I discussed this with @martijnvg in another channel and he will proceed with the implementation. One thing I'm unsure of is this should be a move to a byte-sized based limit as opposed to having the possibility of either (and enforcing both). I lean towards the latter. |
I'm fine with having both but I think a bytes (i.e. memory) based one is enough. I think it's the main motivation to limit it. I struggle to find a good default for the count based one, but it's easy to have sane default (1GB or 512MB) for the bytes. |
This limit is based on the size in bytes of the operations in the write buffer. If this limit is exceeded then no more read operations will be coordinated until the size in bytes of the write buffer has dropped below the configured write buffer size limit. Renamed existing `max_write_buffer_size` to ``max_write_buffer_count` to indicate that limit is count based. Closes elastic#34705
This limit is based on the size in bytes of the operations in the write buffer. If this limit is exceeded then no more read operations will be coordinated until the size in bytes of the write buffer has dropped below the configured write buffer size limit. Renamed existing `max_write_buffer_size` to ``max_write_buffer_count` to indicate that limit is count based. Closes #34705
This limit is based on the size in bytes of the operations in the write buffer. If this limit is exceeded then no more read operations will be coordinated until the size in bytes of the write buffer has dropped below the configured write buffer size limit. Renamed existing `max_write_buffer_size` to ``max_write_buffer_count` to indicate that limit is count based. Closes #34705
This limit is based on the size in bytes of the operations in the write buffer. If this limit is exceeded then no more read operations will be coordinated until the size in bytes of the write buffer has dropped below the configured write buffer size limit. Renamed existing `max_write_buffer_size` to ``max_write_buffer_count` to indicate that limit is count based. Closes #34705
Currently we stop sending read requests if the number of operation waiting to sent as a write bulk is higher than a given number. That was the simplest to implement but it makes it harder to reason about a good value - the ultimate goal of setting a limit is to protect against memory consumption. I think we should move to a byte sized based limit.
/cc @jasontedor @martijnvg
The text was updated successfully, but these errors were encountered: