[resharding] Implement support for flat storage changes

[shreyan-gupta]

Changes

As of current V0 resharding, we do not write to or update flat storage. None of the tests catch this issue as they are not looking at flat storage state and we always fall back to the trie when flat storage doesn't exist for a shard.

We need to implement the following features for flat storage to work with resharding as expected

• Create flat storage for the child shards using flat_storage_manager during resharding initiation.
• Write state to flat storage for the child shards while splitting state
• Update flat storage state and head during catchup.
• Update flat storage state after catchup and while handling split state changes.

Context

Resharding works as follows. Consider we need to start using the child shards at epoch e + 1. Resharding starts at the first block of epoch e.

Resharding

We first call the sequence of functions related to resharding

• chain.build_state_for_split_shards_preprocessing called by client actor
• Chain::build_state_for_split_shards called by sync job actor
  • Here we call trie.add_values_to_split_states function to apply key, value in the parent trie to the child tries.
  • Here we need to write state to flat_storage for child shards.
• chain.build_state_for_split_shards_postprocessing called by client actor

ApplyChunksMode::NotCaughtUp (resharding in progress)

The first step of resharding can take a long time to run and while that is happening we could have advanced multiple blocks. At this point of time the state is NotCaughtUp and split_state_roots is not present.

• Preprocessing
  • While resharding is running we store the changes as ApplySplitStateResultOrStateChanges::StateChangesForSplitStates(state_changes). This happens in the apply_split_state_changes function.
  • This would later be split into state for child shards and applied.
• Postprocessing
  • process_split_state function handles StateChangesForSplitStates and saves it to chain store via chain_store_update.add_state_changes_for_split_states. This would later be drained and applied to the child shards.
  • More specifically, this data is stored in column DBCol::StateChangesForSplitStates.

ApplyChunksMode::CatchingUp

Once resharding is completed, (i.e. split_state_roots is set/present), we can drain the state_changes_for_split_states and apply the changes to the child shards.

• This is done in the process_apply_chunk_result function where we call self.process_split_state with ApplySplitStateResults.

ApplyChunksMode::IsCaughtUp

• After this for every new block, we can call runtime_adapter.apply_update_to_split_states in preprocessing and self.process_split_state in postprocessing.

Tasks

Beta Give feedback

1. [resharding] Include state_changes in ApplySplitStateResult for split_state #9420
   +0
   
2. [resharding] Write flat storage state during resharding #9422
   +0
   
3. [resharding] Update flat storage state and head during process_split_state #9423
   +0
   
4. [resharding] Initialize flat storage during resharding #9424
   +0
   
5. [resharding] Adding unit tests and verifying if flat storage is getting populated as expected

Good to have

Beta Give feedback

1. [resharding] Clean up committing changes from WrappedTrieChanges
2. [flat_storage] Abstract save_flat_state_changes and update_flat_storage_for_shard into a single store commit #9430
   +0
   

   

[shreyan-gupta]

Didn't notice this before, but we have a duplicate issue. Closing the previous one #9207