-
Notifications
You must be signed in to change notification settings - Fork 618
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[resharding] Use state sync snapshots to get flat storage iterator at a precise block #9598
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, nice
… a precise block (#9598) With the current implementation for flat storage iterator, we have the issue that we can not control the block over which we create the iterator. A solution to that is to use the snapshot mechanism from state sync. We can assume that the snapshot would be created on all nodes as of the last block of an epoch. The way we get the correct state for resharding is that we take the snapshot as of the `prev_prev_hash` and append the delta from the `prev_hash` so that the state of the child tries/shards matches that of the last block of prev epoch.
})?; | ||
let flat_storage_iter = | ||
flat_storage_chunk_view.iter_flat_state_entries(None, None).map(|entry| { | ||
let (key, value) = entry.unwrap(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I missed it yesterday but we should properly propagate that error.
let new_shards = next_epoch_shard_layout | ||
.get_split_shard_uids(shard_id) | ||
.ok_or(Error::InvalidShardId(shard_id))?; | ||
let mut state_roots: HashMap<_, _> = | ||
new_shards.iter().map(|shard_uid| (*shard_uid, Trie::EMPTY_ROOT)).collect(); | ||
|
||
// Build the required iterator from flat storage and delta changes. Note that we are |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Can you refactor this to a separate method?
hehe gotcha ;)
… a precise block (#9598) With the current implementation for flat storage iterator, we have the issue that we can not control the block over which we create the iterator. A solution to that is to use the snapshot mechanism from state sync. We can assume that the snapshot would be created on all nodes as of the last block of an epoch. The way we get the correct state for resharding is that we take the snapshot as of the `prev_prev_hash` and append the delta from the `prev_hash` so that the state of the child tries/shards matches that of the last block of prev epoch.
… a precise block (#9598) With the current implementation for flat storage iterator, we have the issue that we can not control the block over which we create the iterator. A solution to that is to use the snapshot mechanism from state sync. We can assume that the snapshot would be created on all nodes as of the last block of an epoch. The way we get the correct state for resharding is that we take the snapshot as of the `prev_prev_hash` and append the delta from the `prev_hash` so that the state of the child tries/shards matches that of the last block of prev epoch.
With the current implementation for flat storage iterator, we have the issue that we can not control the block over which we create the iterator. A solution to that is to use the snapshot mechanism from state sync.
We can assume that the snapshot would be created on all nodes as of the last block of an epoch.
The way we get the correct state for resharding is that we take the snapshot as of the
prev_prev_hash
and append the delta from theprev_hash
so that the state of the child tries/shards matches that of the last block of prev epoch.