-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rework the handling of the ChainConfig.initial_state
during re-genesis
#1209
Comments
xgreenx
added
the
tech-debt
The issue is to improve the current code and make it more clear/generic/reusable/pretty/avoidable.
label
Jun 13, 2023
xgreenx
removed
the
tech-debt
The issue is to improve the current code and make it more clear/generic/reusable/pretty/avoidable.
label
Jun 13, 2023
Closed
Based on the internal conversation, we agreed that the work can be paralyzed in several ways:
|
Also, maybe we need to think about migration. Do we want to make it part of the regenesis, or do we want to have a separate flow to migrate it? |
Merged
xgreenx
added a commit
that referenced
this issue
Mar 5, 2024
Closes #1209 The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. --------- Co-authored-by: Salka1988 <emirsalkicart@gmail.com> Co-authored-by: Brandon Vrooman <brandon.vrooman@fuel.sh> Co-authored-by: segfault_magnet <ahmed.sagdati.ets@gmail.com> Co-authored-by: Ahmed Sagdati <37515857+segfault-magnet@users.noreply.github.com> Co-authored-by: Brandon Kite <brandonkite92@gmail.com> Co-authored-by: Mitchell Turner <james.mitchell.turner@gmail.com> Co-authored-by: Green Baneling <XgreenX9999@gmail.com>
crypto523
added a commit
to crypto523/fuel-core
that referenced
this issue
Oct 7, 2024
Closes FuelLabs/fuel-core#1209 The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. --------- Co-authored-by: Salka1988 <emirsalkicart@gmail.com> Co-authored-by: Brandon Vrooman <brandon.vrooman@fuel.sh> Co-authored-by: segfault_magnet <ahmed.sagdati.ets@gmail.com> Co-authored-by: Ahmed Sagdati <37515857+segfault-magnet@users.noreply.github.com> Co-authored-by: Brandon Kite <brandonkite92@gmail.com> Co-authored-by: Mitchell Turner <james.mitchell.turner@gmail.com> Co-authored-by: Green Baneling <XgreenX9999@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We must process a huge state if we need to re-genesis in the mainnet. It can be so huge that it can not even fit into the RAM. So, we need to re-work the initial state handling.
The obvious solution is to stream the state from the file and process it with chunks. Because this process can be long, we also need to support interruption and continuing from the place where we've stopped. Also, the
initial_state
field in theChainConfig
should be replaced with the final result like genesis hash or genesis metadata instead of storing the whole state in the RAM.The text was updated successfully, but these errors were encountered: