Tracking issue: chunk validator assignment integration #10014
Comments
github-merge-queue bot
pushed a commit
that referenced
this issue
Nov 1, 2023
Implements chunk validator assignment as described in - the docs linked from the [Stateless Validation Organizer](https://docs.google.com/document/d/1gFv3GzPHR5CAX7_X2l5MpuMg7GIjW4Ne8e_3kZLnZQQ/edit#heading=h.d4bbvnvyxo9f) - this [zulip conversation](https://near.zulipchat.com/#narrow/stream/407237-pagoda.2Fcore.2Fstateless-validation/topic/validator.20seat.20assignment) ### The term _seat_ In `nearcore` it is already used: - [`seat_price`](https://github.com/near/nearcore/blob/6f324e84a7a7162956f0b9985094ee146919f5ae/core/primitives/src/epoch_manager.rs#L710-L716) is the minimum stake required to become a validator. - [`NumSeats`](https://github.com/near/nearcore/blob/02f06d1c844296d9b7ea01289ea9e1842f404dd1/core/primitives-core/src/types.rs#L39C1-L40) is the number of seats available for block producers. This might lead to conflicts and confusion with the concept of _seat_ to be introduced for chunk validator assignment. To avoid that, this PR uses the term _mandate_ for now. Accordingly the stake of chunk validators is splitt into _mandates_ which are randomly shuffled and assigned to shards. I’m happy to rename _mandate_ to anything else, though if possible I would try to avoid conflicts with the existing usage of _seat_. ### Overview - Introduces `EpochInfoV4` which contains the epoch’s `ValidatorMandates`. - Adds `EpochInfo::sample_chunk_validators` which allows sampling for a given height. - Module `core::primitives::validator_mandates` encapsulates splitting validator stake into mandates, shuffling them, and assigning them to shards. ### Follow-up concerns As discussed offline ([ref](https://near.zulipchat.com/#narrow/stream/407237-pagoda.2Fcore.2Fstateless-validation/topic/validator.20seat.20assignment/near/396265731)), the following are separate concerns: - Properly integrating stateless validator assignment with the rest of the system. - Dynamically calculating and updating `stake_per_mandate` and `min_mandates_per_shard`. Hence there are some open `TODO`s in the diff, they are marked with #10014.
github-merge-queue bot
pushed a commit
that referenced
this issue
Dec 11, 2023
Introduces a shuffling of shard ids when assigning chunk validators to shards. Without that shuffling it is predictable which shards receive fewer (partial) mandates when they cannot be distributed evenly across shards. See this [comment](mooori/sim-validator-assignment#12 (comment)) which also brings up the idea of shard id shuffling. ### Separate shuffles for full and partial mandate assignment If there was only one shuffling for both full and partial mandates, then the shard(s) that get a full mandate less may also get a partial mandate less. Separate shufflings counter that and ideally (depending on randomness) stake is distributed more evenly as shards that receive a full mandate less may still get the maximum amount of partial mandates. ### Related issue #10014
github-merge-queue bot
pushed a commit
that referenced
this issue
Apr 18, 2024
…rget number of mandates per shard (#11044) This PR completes the remaining tasks in #10014 though in a slightly different way that initially conceived. This PR adds a function to compute the mandate price from a target number of mandates and the distribution of stakes. The function works by looking for the fixed point of a particular function, where this fixed point should exist because it arises from calculating the same value in two different ways. More details on this are found in the comments on the new code. This is a little different than the initial design where we define a minimum number of mandates per shard. Instead we give a target number for the function to attempt to achieve. For many stake distributions it will exactly hit the target, but it is possible that the target cannot be exactly achieved due to the discrete nature of the problem. In this case the function will get as close as possible to the target; even if that means getting a value slightly lower. Importantly, the mandate counting done by the function does not include partial mandates, therefore if the number of whole mandates ends up a little below the target the partial mandates should make up for it to keep the security of the protocol high enough. As part of this PR, the target number of mandates per shard in production is set to 68. This number was chosen based on [some theory calculations](https://near.zulipchat.com/#narrow/stream/407237-core.2Fstateless-validation/topic/validator.20seat.20assignment/near/430080139). In a future PR this number should probably be extracted out as a genesis config parameter, but I think it's ok for now (stateless validation MVP).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Tracking issue for the integration of chunk validator assignment (#9983) with the rest of the system:
Determine an initial value for
min_mandates_per_shard.ValidatorMandatesConfig.Determine an initial value for
stake_per_mandate.num_shards * min_mandates_per_shard.ValidatorMandatesConfig.Lower
stake_per_mandatedynamically when there are not enough mandates for all shards with the current set of validators.Introduce a protocol feature which determines whether
EpochInfo::new()should returnEpochInfo::V3orEpochInfo::V4(ref).Shuffle shard ids to avoid a bias towards smaller shard ids (ref).
The text was updated successfully, but these errors were encountered: