Store block range roots in db #1650

Alenar · 2024-04-22T16:18:43Z

Content

This PR add block range root (aka a BlockRange and it's associated transactions merkle root) computation and storage.

This step is added to the CardanoTransactionImporter of signer & aggregator nodes after the import of the transactions.

Other changes:

mithril-common: Change and rename the TransactionParser into a BlockScanner that returns a dedicated type ScannedBlock that can be converted into a list of transactions. This simplify the next refactor to make it stream the blocks in order to lessen the scanning footprint (Stream import of Cardano transactions #1646).
mithril-persistence: Add a system to add quickly a get_all method to a type that implement the Provider trait. You can now implement the GetAllCondition trait to the entity of the provider to automatically derive the GetAllProvider trait that define the get_all method.

Pre-submit checklist

Branch
- Tests are provided (if possible)
- Crates versions are updated (if relevant)
- CHANGELOG file is updated (if relevant)
- Commit sequence broadly makes sense
- Key commits have useful messages
PR
- No clippy warnings in the CI
- Self-reviewed the diff
- Useful pull request description
- Reviewer requested
Documentation
- Update README file (if relevant)
- Update documentation website (if relevant)
- Add dev blog post (if relevant)

Issue(s)

Closes #1633

github-actions · 2024-04-22T16:25:07Z

Test Results

3 files ± 0 43 suites ±0 8m 16s ⏱️ -15s
972 tests +44 972 ✅ +44 0 💤 ±0 0 ❌ ±0
1 066 runs +44 1 066 ✅ +44 0 💤 ±0 0 ❌ ±0

Results for commit b0decd8. ± Comparison against base commit 86c172c.

This pull request removes 15 and adds 59 tests. Note that renamed tests count towards both.

mithril-aggregator ‑ database::repository::cardano_transaction_repository::tests::repository_store_transactions_doesnt_erase_existing_data
mithril-aggregator ‑ services::cardano_transactions_importer::tests::if_all_half_are_stored_the_other_half_is_parsed_and_stored
mithril-aggregator ‑ services::cardano_transactions_importer::tests::if_all_stored_nothing_is_parsed_and_stored
mithril-aggregator ‑ services::cardano_transactions_importer::tests::importing_twice_starting_with_nothing_in_a_real_db_should_yield_the_transactions_in_same_order
mithril-common ‑ cardano_transaction_parser::tests::test_instantiate_parser_with_allow_unparsable_block_should_log_warning
mithril-common ‑ cardano_transaction_parser::tests::test_instantiate_parser_without_allow_unparsable_block_should_not_log_warning
mithril-common ‑ cardano_transaction_parser::tests::test_parse_expected_number_of_transactions
mithril-common ‑ cardano_transaction_parser::tests::test_parse_from_lower_bound_until_upper_bound
mithril-common ‑ cardano_transaction_parser::tests::test_parse_should_error_with_unparsable_block_format
mithril-common ‑ cardano_transaction_parser::tests::test_parse_should_log_error_with_unparsable_block_format
…

mithril-aggregator ‑ database::record::block_range_root::tests::hydrate_fail_if_invalid_block_range_in_row
mithril-aggregator ‑ database::record::block_range_root::tests::hydrate_succeed_if_valid_block_range_in_row
mithril-aggregator ‑ database::record::interval_without_block_range_root::tests::db_with_both_block_range_and_transactions_yield_a_range
mithril-aggregator ‑ database::record::interval_without_block_range_root::tests::db_with_last_transaction_block_number_below_the_end_of_the_last_block_range_yield_an_error
mithril-aggregator ‑ database::record::interval_without_block_range_root::tests::db_with_last_transaction_block_number_right_at_the_end_of_the_last_block_range_yield_none
mithril-aggregator ‑ database::record::interval_without_block_range_root::tests::db_with_only_block_range_and_no_transactions_yield_none
mithril-aggregator ‑ database::record::interval_without_block_range_root::tests::db_with_only_transactions_yield_a_range
mithril-aggregator ‑ database::record::interval_without_block_range_root::tests::db_without_block_range_nor_transactions_yield_none
mithril-aggregator ‑ database::repository::cardano_transaction_repository::tests::repository_get_block_interval_without_block_range_root
mithril-aggregator ‑ database::repository::cardano_transaction_repository::tests::repository_get_transactions_in_range_blocks
…

♻️ This comment has been updated with latest results.

mithril-aggregator/src/database/provider/block_range_root/get_block_range_root.rs

internal/mithril-persistence/src/sqlite/condition.rs

mithril-aggregator/src/database/record/block_range_root.rs

mithril-aggregator/src/database/record/interval_without_block_range_root.rs

mithril-aggregator/src/services/cardano_transactions_importer.rs

mithril-common/src/entities/block_range.rs

dlachaume

LGTM overall, just need to clarify @jpraynaud's last comments.

This reduce the 'move' boilerplates needed to work with the mocks.

Since we will change it's main method signature to retrieve a list of blocks instead of a list of transactions.

…ransactions

…root

Automatically implemented for all provider if the Provider implement a new trait `GetAllCondition`.

This allow to lessen the compling between the tests and the implementations, making the code easier to changes while keeping the tests safety net.

By only loading a block range worth of transactions from db instead of loading them all at once.

To uniformise use of ranges since `BlockRange` use a Range internaly.

Two transactions that we get from the chain may have a gap of block numbers that is larger than a block range. In this case the importer still tried to compute the merkle root for the block ranges that did not contains any transaction, leading to an error.

It raised an retryable error that the state machine tried immediatly to retry leading to a loop. + Refactor the import_block_range method to flatten it's code, making it more readble. + Add a log that tell which BlockRangeRoots the importer will try to compute.

…ent state Before it only yielded that no work was needed

* start & end must be multiples of BlockRange::Length * start & end interval must be equal to BlockRange::Length

Instead of rowid, this is more consistent accross nodes.

Previous naive implementation was failing with very large intervals (larger than several millions) since the whole interval was loaded in memory. In order to avoid this side effect an iterator type, `BlockRangesSequence`, is now yielded. Allowing to lazely iterate a sequence of block ranges.

To be consistent with all usages or ranges for blocks, this remove the last usage of a range inclusive to only use non-inclusive range. The last case of RangeInclusive was a test that checked that an inverval is contained in a block ranges sequence. To do so it add a `contains(range)` to `BlockRangesSequence`.

…nge root * This computation forgot to handle the case when the last stored transaction is included at the end of the last stored block range root. When this case was seen it yielded an error instead of saying that there was no work to be done. record it derives from. This allow to write simpler tests.

* Mithril-aggregator from `0.4.62` to `0.4.63` * Mithril-common from `0.3.33` to `0.3.34` * Mithril-signer from `0.2.126` to `0.2.127` * Mithril-persitence from `0.1.6` to `0.1.7`

Alenar requested review from jpraynaud and dlachaume April 22, 2024 16:18

Alenar changed the title ~~Ensemble/1633/store block ranges in db~~ Store block ranges in db Apr 22, 2024

Alenar changed the title ~~Store block ranges in db~~ Store block range roots in db Apr 22, 2024

Alenar force-pushed the ensemble/1633/store-block-ranges-in-db branch 2 times, most recently from 06e012f to 9ca6619 Compare April 22, 2024 18:34

Alenar temporarily deployed to testing-preview April 22, 2024 18:41 — with GitHub Actions Inactive

Alenar temporarily deployed to testing-sanchonet April 22, 2024 18:41 — with GitHub Actions Inactive

Alenar temporarily deployed to testing-preview April 24, 2024 17:04 — with GitHub Actions Inactive

Alenar temporarily deployed to testing-sanchonet April 24, 2024 17:04 — with GitHub Actions Inactive

Alenar mentioned this pull request Apr 24, 2024

Stream blocks during import #1652

Merged

12 tasks

jpraynaud reviewed Apr 25, 2024

View reviewed changes

dlachaume approved these changes Apr 25, 2024

View reviewed changes

Alenar force-pushed the ensemble/1633/store-block-ranges-in-db branch 2 times, most recently from 835021a to 409530d Compare April 26, 2024 07:37

Alenar added 14 commits April 26, 2024 11:37

Use FnOnce to init mocks in tx importer test

69f1ed8

This reduce the 'move' boilerplates needed to work with the mocks.

Rename TransactionParser to BlockScanner

3751bb7

Since we will change it's main method signature to retrieve a list of blocks instead of a list of transactions.

Change BlockScanner to return a list of Blocks instead of a list of T…

86c92e1

…ransactions

Make signer TransactionImporter store block ranges with their merkle …

0c72a96

…root

Compute block range list to compute from the scanned blocks

61c817f

Add start and all_ranges_in to BlockRange

dcbb337

Importer can now compute & store block range merkle roots

defda78

Add a test for merkle root computation on Importer

b6a6939

Simplify get_transactions_between signature by using a range

150f5eb

Implement get_transactions_between on CardanoTransactionRepository

f17312f

Add get_all to database provider

8e91244

Automatically implemented for all provider if the Provider implement a new trait `GetAllCondition`.

Implement database layer for block_range_root

f2f37bf

Generalize usage of the common GetAllProvider

6a485e5

Refactor transactions importer test to use a real in memory db

eca2aa1

This allow to lessen the compling between the tests and the implementations, making the code easier to changes while keeping the tests safety net.

Alenar added 12 commits April 26, 2024 11:37

Reduce compute block range roots footprints

c7c0f67

By only loading a block range worth of transactions from db instead of loading them all at once.

Use Range instead of RangeInclusive in functions signatures

b6e0cb8

To uniformise use of ranges since `BlockRange` use a Range internaly.

Synchronise importer changes from signer to aggregator

799f046

Warn log adjustment after self review

afe0182

PR review: Style & naming fix from PR

112d43b

PR review: make block range root interval retrieval fail on inconsist…

a7a7720

…ent state Before it only yielded that no work was needed

PR Review: make BlockRangeRootRecor::hydrate fail if range is invalid

7cdf97b

* start & end must be multiples of BlockRange::Length * start & end interval must be equal to BlockRange::Length

PR Review: order db retrieved block range root by start & end

23dfabc

Instead of rowid, this is more consistent accross nodes.

Alenar had a problem deploying to testing-preview April 26, 2024 09:39 — with GitHub Actions Error

Alenar temporarily deployed to testing-sanchonet April 26, 2024 09:39 — with GitHub Actions Inactive

Alenar force-pushed the ensemble/1633/store-block-ranges-in-db branch from b72b1df to 01df776 Compare April 26, 2024 09:39

Alenar added 2 commits April 26, 2024 13:53

Upgrade crates versions

b0decd8

* Mithril-aggregator from `0.4.62` to `0.4.63` * Mithril-common from `0.3.33` to `0.3.34` * Mithril-signer from `0.2.126` to `0.2.127` * Mithril-persitence from `0.1.6` to `0.1.7`

Alenar force-pushed the ensemble/1633/store-block-ranges-in-db branch from 01df776 to b0decd8 Compare April 26, 2024 11:53

Alenar temporarily deployed to testing-preview April 26, 2024 12:00 — with GitHub Actions Inactive

Alenar temporarily deployed to testing-sanchonet April 26, 2024 12:00 — with GitHub Actions Inactive

Alenar merged commit c0a7b67 into main Apr 26, 2024
41 of 43 checks passed

Alenar deleted the ensemble/1633/store-block-ranges-in-db branch April 26, 2024 12:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store block range roots in db #1650

Store block range roots in db #1650

Alenar commented Apr 22, 2024 •

edited

Loading

github-actions bot commented Apr 22, 2024 •

edited

Loading

dlachaume left a comment

Store block range roots in db #1650

Store block range roots in db #1650

Conversation

Alenar commented Apr 22, 2024 • edited Loading

Content

Pre-submit checklist

Issue(s)

github-actions bot commented Apr 22, 2024 • edited Loading

Test Results

dlachaume left a comment

Choose a reason for hiding this comment

Alenar commented Apr 22, 2024 •

edited

Loading

github-actions bot commented Apr 22, 2024 •

edited

Loading