Feature/new fti event #1770

Voxelot · 2024-03-21T00:36:51Z

Adds a new relayer event type in support of forced transactions, assuming the following ABI from L1:

[
    event MessageSent(bytes32 indexed sender, bytes32 indexed recipient, uint256 indexed nonce, uint64 amount, bytes data)
    event Transaction(uint64 max_gas, bytes canonically_serialized_tx)
]

Specifically, this PR adds:

Parsing FTI logs from ethereum
Saving parsed FTI events to the relayer events table
Reshuffles the message entity in fuel-core-types to be grouped under a relayer module, which also includes the new entity for RelayedTransaction's (this may cause a lot of breaking changes for the regenesis workstream, so if this seems like more trouble than it's worth I can put things back to how they were)

# Conflicts: # CHANGELOG.md # bin/fuel-core/src/cli/snapshot.rs # crates/fuel-core/src/service/genesis.rs # crates/storage/src/tables.rs

crates/services/relayer/src/mock_db.rs

crates/types/src/entities/relayer/transaction.rs

crates/services/relayer/src/log.rs

Co-authored-by: Green Baneling <XgreenX9999@gmail.com>

# Conflicts: # crates/types/src/services/relayer.rs

crates/fuel-core/src/executor.rs

crates/services/relayer/tests/integration.rs

crates/types/src/entities/relayer/transaction.rs

crates/types/src/services/relayer.rs

crates/types/src/entities/relayer/transaction.rs

bvrooman · 2024-03-22T22:43:17Z

crates/services/relayer/src/log.rs

@@ -97,6 +122,35 @@ impl TryFrom<&Log> for EthEventLog {
                    ),
                })
            }
+            n if n == *config::ETH_FORCED_TX => {
+                if log.topics.len() != 1 {


The 1 is a bit of a magic number. I looked up solidity topics and found:

topic[0] always refers to the hash of the hash of the event itself, and can have up to 3 indexed arguments, which will each be reflected in the topics.

Maybe it would be worth using a constant with a comment about why we use this value?

added some comments for clarity

MitchTurner

Nice. LGTM. I don't have an opinion on the namespacing changes. Might want to check with someone who does before merging.

Co-authored-by: Brandon Vrooman <brandon.vrooman@fuel.sh>

@Dentosal

## [Version 0.24.0] ### Added - [#1786](#1786): Regenesis now includes off-chain tables. - [#1716](#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](#1770): Add the new L1 event type for forced transactions. - [#1767](#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](#1747): The DA block height is now included in the genesis state. - [#1740](#1740): Remove optional fields from genesis configs - [#1737](#1737): Remove temporary tables for calculating roots during genesis. - [#1731](#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](#1779): Modify Relayer service to order Events from L1 by block index - [#1783](#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after #1664. - [#1693](#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in #1731 * Regenesis support by @MujkicA in #1693 * Remove genesis temp tables by @MujkicA in #1737 * Remove optional fields from configs by @MujkicA in #1740 * Weekly `cargo update` by @github-actions in #1745 * Regenesis should also store da block height by @xgreenx in #1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in #1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in #1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in #1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in #1757 * Weekly `cargo update` by @github-actions in #1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in #1760 * Adjustments to the upcoming testnet configs by @xgreenx in #1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in #1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in #1768 * Keep record of events from L1 in Block Header by @MitchTurner in #1769 * Feature/new fti event by @Voxelot in #1770 * Forkless state transition with upgradable WASM executor by @xgreenx in #1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in #1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in #1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in #1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in #1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in #1786 **Full Changelog**: v0.23.0...v0.24.0

@Dentosal

## Version v0.24.0 ### Added - [#1786](#1786): Regenesis now includes off-chain tables. - [#1716](#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](#1770): Add the new L1 event type for forced transactions. - [#1767](#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](#1747): The DA block height is now included in the genesis state. - [#1740](#1740): Remove optional fields from genesis configs - [#1737](#1737): Remove temporary tables for calculating roots during genesis. - [#1731](#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](#1779): Modify Relayer service to order Events from L1 by block index - [#1783](#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after #1664. - [#1693](#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in #1731 * Regenesis support by @MujkicA in #1693 * Remove genesis temp tables by @MujkicA in #1737 * Remove optional fields from configs by @MujkicA in #1740 * Weekly `cargo update` by @github-actions in #1745 * Regenesis should also store da block height by @xgreenx in #1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in #1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in #1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in #1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in #1757 * Weekly `cargo update` by @github-actions in #1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in #1760 * Adjustments to the upcoming testnet configs by @xgreenx in #1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in #1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in #1768 * Keep record of events from L1 in Block Header by @MitchTurner in #1769 * Feature/new fti event by @Voxelot in #1770 * Forkless state transition with upgradable WASM executor by @xgreenx in #1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in #1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in #1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in #1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in #1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in #1786 * Release v0.24.0 by @xgreenx in #1791 * Moved chain specification into `fuel-core-bin` crate by @xgreenx in #1792 **Full Changelog**: v0.23.0...v0.24.1

@Dentosal

## [Version 0.24.0] ### Added - [#1786](FuelLabs/fuel-core#1786): Regenesis now includes off-chain tables. - [#1716](FuelLabs/fuel-core#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](FuelLabs/fuel-core#1770): Add the new L1 event type for forced transactions. - [#1767](FuelLabs/fuel-core#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](FuelLabs/fuel-core#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](FuelLabs/fuel-core#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](FuelLabs/fuel-core#1747): The DA block height is now included in the genesis state. - [#1740](FuelLabs/fuel-core#1740): Remove optional fields from genesis configs - [#1737](FuelLabs/fuel-core#1737): Remove temporary tables for calculating roots during genesis. - [#1731](FuelLabs/fuel-core#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](FuelLabs/fuel-core#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](FuelLabs/fuel-core#1779): Modify Relayer service to order Events from L1 by block index - [#1783](FuelLabs/fuel-core#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](FuelLabs/fuel-core#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](FuelLabs/fuel-core#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](FuelLabs/fuel-core#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](FuelLabs/fuel-core#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](FuelLabs/fuel-core#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](FuelLabs/fuel-core@7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](FuelLabs/fuel-core@5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after FuelLabs/fuel-core#1664. - [#1693](FuelLabs/fuel-core#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](FuelLabs/fuel-core#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in FuelLabs/fuel-core#1731 * Regenesis support by @MujkicA in FuelLabs/fuel-core#1693 * Remove genesis temp tables by @MujkicA in FuelLabs/fuel-core#1737 * Remove optional fields from configs by @MujkicA in FuelLabs/fuel-core#1740 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1745 * Regenesis should also store da block height by @xgreenx in FuelLabs/fuel-core#1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in FuelLabs/fuel-core#1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in FuelLabs/fuel-core#1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in FuelLabs/fuel-core#1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in FuelLabs/fuel-core#1757 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in FuelLabs/fuel-core#1760 * Adjustments to the upcoming testnet configs by @xgreenx in FuelLabs/fuel-core#1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in FuelLabs/fuel-core#1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in FuelLabs/fuel-core#1768 * Keep record of events from L1 in Block Header by @MitchTurner in FuelLabs/fuel-core#1769 * Feature/new fti event by @Voxelot in FuelLabs/fuel-core#1770 * Forkless state transition with upgradable WASM executor by @xgreenx in FuelLabs/fuel-core#1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in FuelLabs/fuel-core#1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in FuelLabs/fuel-core#1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in FuelLabs/fuel-core#1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in FuelLabs/fuel-core#1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in FuelLabs/fuel-core#1786 **Full Changelog**: FuelLabs/fuel-core@v0.23.0...v0.24.0

@Dentosal

## Version v0.24.0 ### Added - [#1786](FuelLabs/fuel-core#1786): Regenesis now includes off-chain tables. - [#1716](FuelLabs/fuel-core#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](FuelLabs/fuel-core#1770): Add the new L1 event type for forced transactions. - [#1767](FuelLabs/fuel-core#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](FuelLabs/fuel-core#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](FuelLabs/fuel-core#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](FuelLabs/fuel-core#1747): The DA block height is now included in the genesis state. - [#1740](FuelLabs/fuel-core#1740): Remove optional fields from genesis configs - [#1737](FuelLabs/fuel-core#1737): Remove temporary tables for calculating roots during genesis. - [#1731](FuelLabs/fuel-core#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](FuelLabs/fuel-core#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](FuelLabs/fuel-core#1779): Modify Relayer service to order Events from L1 by block index - [#1783](FuelLabs/fuel-core#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](FuelLabs/fuel-core#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](FuelLabs/fuel-core#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](FuelLabs/fuel-core#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](FuelLabs/fuel-core#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](FuelLabs/fuel-core#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](FuelLabs/fuel-core@7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](FuelLabs/fuel-core@5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after FuelLabs/fuel-core#1664. - [#1693](FuelLabs/fuel-core#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](FuelLabs/fuel-core#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in FuelLabs/fuel-core#1731 * Regenesis support by @MujkicA in FuelLabs/fuel-core#1693 * Remove genesis temp tables by @MujkicA in FuelLabs/fuel-core#1737 * Remove optional fields from configs by @MujkicA in FuelLabs/fuel-core#1740 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1745 * Regenesis should also store da block height by @xgreenx in FuelLabs/fuel-core#1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in FuelLabs/fuel-core#1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in FuelLabs/fuel-core#1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in FuelLabs/fuel-core#1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in FuelLabs/fuel-core#1757 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in FuelLabs/fuel-core#1760 * Adjustments to the upcoming testnet configs by @xgreenx in FuelLabs/fuel-core#1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in FuelLabs/fuel-core#1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in FuelLabs/fuel-core#1768 * Keep record of events from L1 in Block Header by @MitchTurner in FuelLabs/fuel-core#1769 * Feature/new fti event by @Voxelot in FuelLabs/fuel-core#1770 * Forkless state transition with upgradable WASM executor by @xgreenx in FuelLabs/fuel-core#1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in FuelLabs/fuel-core#1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in FuelLabs/fuel-core#1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in FuelLabs/fuel-core#1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in FuelLabs/fuel-core#1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in FuelLabs/fuel-core#1786 * Release v0.24.0 by @xgreenx in FuelLabs/fuel-core#1791 * Moved chain specification into `fuel-core-bin` crate by @xgreenx in FuelLabs/fuel-core#1792 **Full Changelog**: FuelLabs/fuel-core@v0.23.0...v0.24.1

Voxelot added 13 commits March 19, 2024 01:12

add l1 event type for relayed forced transactions

16abcd2

fix refactoring

dff7844

newtype helpers

580166d

add test for getting transactions

e435eb9

add todo

bba8e43

fix comment

335733e

remove commented out code

3943f67

add some extra testing

6364956

add transaction events to real storage tests

a69fcde

CHANGELOG.md

64bfac6

Merge branch 'master' into feature/new-fti-event

17c76fe

# Conflicts: # CHANGELOG.md # bin/fuel-core/src/cli/snapshot.rs # crates/fuel-core/src/service/genesis.rs # crates/storage/src/tables.rs

CHANGELOG.md

4a70f67

minor cleanup

2f66d45

Voxelot marked this pull request as ready for review March 21, 2024 01:22

fix merge conflict with master

2b2b9fa

Voxelot requested a review from a team March 21, 2024 02:45

Dentosal reviewed Mar 21, 2024

View reviewed changes

crates/services/relayer/src/mock_db.rs Show resolved Hide resolved

xgreenx reviewed Mar 22, 2024

View reviewed changes

crates/types/src/entities/relayer/transaction.rs Outdated Show resolved Hide resolved

crates/services/relayer/src/log.rs Outdated Show resolved Hide resolved

Voxelot and others added 4 commits March 22, 2024 07:55

Update crates/services/relayer/src/log.rs

131aafc

Co-authored-by: Green Baneling <XgreenX9999@gmail.com>

remove TODO

c916565

Merge branch 'master' into feature/new-fti-event

2c827e0

# Conflicts: # crates/types/src/services/relayer.rs

add relayed transactions to executor test for merkle root

8deb932

Voxelot requested review from Dentosal and xgreenx March 22, 2024 17:51

MitchTurner reviewed Mar 22, 2024

View reviewed changes

crates/fuel-core/src/executor.rs Show resolved Hide resolved

MitchTurner reviewed Mar 22, 2024

View reviewed changes

crates/services/relayer/tests/integration.rs Outdated Show resolved Hide resolved

MitchTurner reviewed Mar 22, 2024

View reviewed changes

crates/types/src/entities/relayer/transaction.rs Outdated Show resolved Hide resolved

bvrooman reviewed Mar 22, 2024

View reviewed changes

refactor tests

1a0413d

MitchTurner previously approved these changes Mar 22, 2024

View reviewed changes

Update crates/types/src/services/relayer.rs

3b7e397

Co-authored-by: Brandon Vrooman <brandon.vrooman@fuel.sh>

Voxelot dismissed MitchTurner’s stale review via 3b7e397 March 22, 2024 23:18

Voxelot added 2 commits March 22, 2024 16:27

explain magic topic numbers

a8bf11f

put helper functions under test-helpers flag

c28f13f

xgreenx previously approved these changes Mar 22, 2024

View reviewed changes

fix overly restrictive feature flags

b16688a

Voxelot dismissed xgreenx’s stale review via b16688a March 22, 2024 23:33

xgreenx approved these changes Mar 22, 2024

View reviewed changes

Voxelot merged commit fb79894 into master Mar 22, 2024
30 checks passed

Voxelot deleted the feature/new-fti-event branch March 22, 2024 23:41

xgreenx assigned Voxelot Mar 25, 2024

xgreenx mentioned this pull request Mar 30, 2024

Release v0.24.0 #1791

Merged

xgreenx mentioned this pull request Mar 30, 2024

Release v0.24.1 #1793

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/new fti event #1770

Feature/new fti event #1770

Voxelot commented Mar 21, 2024 •

edited

Loading

bvrooman Mar 22, 2024

Voxelot Mar 22, 2024

MitchTurner left a comment

Feature/new fti event #1770

Feature/new fti event #1770

Conversation

Voxelot commented Mar 21, 2024 • edited Loading

bvrooman Mar 22, 2024

Choose a reason for hiding this comment

Voxelot Mar 22, 2024

Choose a reason for hiding this comment

MitchTurner left a comment

Choose a reason for hiding this comment

Voxelot commented Mar 21, 2024 •

edited

Loading