-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
admin: methods to select replicas for log stream #393
Closed
4 of 6 tasks
Comments
ijsong
added a commit
that referenced
this issue
Apr 5, 2023
It adds a new method `Name() string` to the replica selector interface `internal/admin.(ReplicaSelector)`. We can set a unique name for each replica selector, and therefore, we can use a necessary replica selection algorithm by specifying the name. Updates #393
ijsong
added a commit
that referenced
this issue
Apr 5, 2023
It adds a new replica selector, `randomReplicaSelector`. It chooses storage nodes and paths randomly. Updates #393
ijsong
added a commit
that referenced
this issue
Apr 5, 2023
This PR adds a new replica selector based on the Least Frequently Used (LFU) algorithm. `lfuReplicaSelector` selects each replica's storage node and data path, giving preference to those with fewer assigned replicas. All replica selectors, including `lfuReplicaSelector`, are stateless, meaning they don't keep existing log stream topology. When `Select` is invoked, `lfuReplicaSelector` collects used counters for each storage node and path. Although it seems to be inefficient, it is simple as well as fault-tolerant. To select the least used storage nodes and paths, `lfuReplicaSelector` runs as follows: - Fetch cluster metadata from the `ClusterMetadataView`. - Increase the used counters for each storage node and path assigned to each log stream. - Sort counters. - Choose the least used storage nodes and paths. Updates #393
This was
linked to
pull requests
Apr 5, 2023
ijsong
added a commit
that referenced
this issue
Apr 6, 2023
It adds a new method `Name() string` to the replica selector interface `internal/admin.(ReplicaSelector)`. We can set a unique name for each replica selector, and therefore, we can use a necessary replica selection algorithm by specifying the name. Updates #393
ijsong
added a commit
that referenced
this issue
Apr 6, 2023
It adds a new replica selector, `randomReplicaSelector`. It chooses storage nodes and paths randomly. Updates #393
ijsong
added a commit
that referenced
this issue
Apr 6, 2023
This PR adds a new replica selector based on the Least Frequently Used (LFU) algorithm. `lfuReplicaSelector` selects each replica's storage node and data path, giving preference to those with fewer assigned replicas. All replica selectors, including `lfuReplicaSelector`, are stateless, meaning they don't keep existing log stream topology. When `Select` is invoked, `lfuReplicaSelector` collects used counters for each storage node and path. Although it seems to be inefficient, it is simple as well as fault-tolerant. To select the least used storage nodes and paths, `lfuReplicaSelector` runs as follows: - Fetch cluster metadata from the `ClusterMetadataView`. - Increase the used counters for each storage node and path assigned to each log stream. - Sort counters. - Choose the least used storage nodes and paths. Updates #393
ijsong
added a commit
that referenced
this issue
Apr 6, 2023
It added a new flag, `replica-selector`, to the varlogadm. The flag sets the global default replica selector. Updates #393
ijsong
added a commit
that referenced
this issue
Apr 6, 2023
It adds a new method `Name() string` to the replica selector interface `internal/admin.(ReplicaSelector)`. We can set a unique name for each replica selector, and therefore, we can use a necessary replica selection algorithm by specifying the name. Updates #393
ijsong
added a commit
that referenced
this issue
Apr 6, 2023
It adds a new replica selector, `randomReplicaSelector`. It chooses storage nodes and paths randomly. Updates #393
ijsong
added a commit
that referenced
this issue
Apr 6, 2023
This PR adds a new replica selector based on the Least Frequently Used (LFU) algorithm. `lfuReplicaSelector` selects each replica's storage node and data path, giving preference to those with fewer assigned replicas. All replica selectors, including `lfuReplicaSelector`, are stateless, meaning they don't keep existing log stream topology. When `Select` is invoked, `lfuReplicaSelector` collects used counters for each storage node and path. Although it seems to be inefficient, it is simple as well as fault-tolerant. To select the least used storage nodes and paths, `lfuReplicaSelector` runs as follows: - Fetch cluster metadata from the `ClusterMetadataView`. - Increase the used counters for each storage node and path assigned to each log stream. - Sort counters. - Choose the least used storage nodes and paths. Updates #393
ijsong
added a commit
that referenced
this issue
Apr 6, 2023
It added a new flag, `replica-selector`, to the varlogadm. The flag sets the global default replica selector. Updates #393
ijsong
added a commit
that referenced
this issue
Apr 6, 2023
It added a new flag, `replica-selector`, to the varlogadm. The flag sets the global default replica selector. Updates #393
ijsong
added a commit
that referenced
this issue
Apr 6, 2023
It added a new flag, `replica-selector`, to the varlogadm. The flag sets the global default replica selector. Updates #393
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When the admin makes a log stream, it can select replicas automatically if a client does not send the log stream's topology. The admin provides the interface
ReplicaSelector
to support various methods of choosing replicas.Currently, the admin provides
balancedReplicaSelector
, which implementsReplicaSelector
; however, it does not work well because it is unnecessarily too complex. Therefore we have to revisit it.Goals:
Implementation plan:
Name
to theReplicaSelector
interface, which returns a unique name of each replica selector. The name can define the replica selector that the admin server uses. feat(admin): add a name to the replica selector #394random
. When there are many storage nodes, replicas of a log stream can be chosen randomly. feat(admin): random replica selector #395balancedReplicaSelector
. It will deploy replicas across storage nodes to balance the number of replicas per storage node and the number of primary replicas per storage node. feat(admin): LFU replica selector #396replica-selector
to the varlogadm #402The text was updated successfully, but these errors were encountered: