Introduce new configuration for limiting replica's local replication buffer during dual-channel replication sync

[naglera]

This PR introduces a new configuration option, replicas-dual-channel-buffer-limit, to better control the size of the replica's local replication buffer during dual-channel replication sync. This configuration will allow for more precise adjustment of the buffer size, ensuring that the sync process can succeed while optimizing resource usage on the replica.

Motivation:
Currently, during dual-channel replication sync, the replica's local replication buffer size is limited by the replica's client_obuf_limits configuration. However, this configuration is not specifically designed for dual-channel replication and may impose unnecessary restrictions on the buffer size, leading to suboptimal performance or even sync failures in certain scenarios.

By introducing a dedicated configuration option for the local replication buffer limit, we can decouple it from the output buffer limit and tailor it specifically for the dual-channel replication sync process. This will enable more accurate estimation and adjustment of the buffer size based on factors such as the total free space on the replica's machine and the estimated snapshot size.

Proposed Changes:

1. Add a new configuration option, replicas-dual-channel-buffer-limit, to the Redis configuration file.
2. Modify the dual-channel replication sync process to respect the new configuration option when setting the local replication buffer size on the replica.

Future Enhancements:
In the future, we will be able to enhance the replicas-dual-channel-buffer-limit configuration by introducing dynamic adjustment capabilities. This will allow the buffer size to be automatically adjusted based on the total free space on the replica's machine and the estimated snapshot size, further optimizing resource usage and sync performance.

[madolson]

Do you have more of a concrete use case that you are mentioning? I was also originally a fan of using the client output buffer limit to keep the configuration simpler. Is there a specific need for having this be decoupled, the motivation feels a bit vague to me.

[naglera]

The client output buffer limit is primarily designed to control the size of the output buffer for client connections. This configuration is intended to prevent a single client from consuming an excessive amount of memory, which could potentially impact the overall performance of the server.

However, during the dual-channel replication sync, the replica's main responsibility is to synchronize its data with the primary instance. It is not serving any client connections or handling read/write requests. Instead, the replica's entire job is to receive a large amount of data from the primary. The only consideration for the local replication buffer limit during sync is ensuring that there is sufficient space to accommodate the entire snapshot.

Dedicated configuration allows us to fine-tune the buffer size specifically for the dual-channel replication sync process, taking into account the unique requirements and constraints of this operation.

previosly decussed here: #60 (comment)

[madolson]

Preference from the core discussion was that we preferred no-config since the buffer size is not the client output buffer. There is a general preference that this configuration should be "smart" and grow automatically instead of having a fixed hard limit. A config could be added if we really see a need for it.

[naglera]

I agree that a "smart" configuration approach would be the best option for managing the replica's local replication buffer size during dual-channel replication sync.

The key to implementing a smart configuration would be the ability to accurately predict the RDB size before its creation. If we can reliably estimate the RDB size, we can then use that information to dynamically allocate an appropriate replication buffer size.

Is there an existing action item or plan to implement RDB size prediction capabilities? If not, would it be possible to create one? Having this functionality would enable us to move forward with a smart, dynamic buffer size configuration for dual-channel replication sync.

[madolson]

Is there an existing action item or plan to implement RDB size prediction capabilities? If not, would it be possible to create one? Having this functionality would enable us to move forward with a smart, dynamic buffer size configuration for dual-channel replication sync.

No, would you mind starting an issue to document the behavior?