Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix unneeded rescan when importing transactions in chunk #2035

Merged
merged 6 commits into from
Oct 22, 2024

Conversation

Alenar
Copy link
Collaborator

@Alenar Alenar commented Oct 21, 2024

Content

This PR fix the issue described in #2021.

To do so the way that the importer obtains its starting point change:

  • Before it would compute it by getting data from the highest stored transactions in database, this caused the issue since in test networks there may be huge ranges of blocks without transactions.
  • Now if keep stored the last ChainPoint that was polled by its block streamer after each import, so it can be used as the next import start point, solving the issue.

Curated logs from the e2e before the change

The same start block number is used repeatedly

{ "msg": "Retrieving Cardano transactions between block_number '0' and '150'" }
{ "msg": "Retrieving Cardano transactions between block_number '0' and '300'" }
{ "msg": "Retrieving Cardano transactions between block_number '254' and '419'" }
{ "msg": "Retrieving Cardano transactions between block_number '341' and '434'" }
{ "msg": "Retrieving Cardano transactions between block_number '341' and '449'" }
{ "msg": "Retrieving Cardano transactions between block_number '341' and '449'" }
{ "msg": "Retrieving Cardano transactions between block_number '341' and '464'" }

Curated logs from the e2e after the change

No start block number is used more than once

{ "msg": "Retrieving Cardano transactions between block_number '0' and '150'" }
{ "msg": "Retrieving Cardano transactions between block_number '150' and '300'" }
{ "msg": "Retrieving Cardano transactions between block_number '300' and '450'" }
{ "msg": "Retrieving Cardano transactions between block_number '450' and '569'" }
{ "msg": "Retrieving Cardano transactions between block_number '569' and '584'" }
{ "msg": "Retrieving Cardano transactions between block_number '584' and '599'" }
{ "msg": "Retrieving Cardano transactions between block_number '599' and '614'" }

Pre-submit checklist

  • Branch
    • Tests are provided (if possible)
    • Crates versions are updated (if relevant)
    • CHANGELOG file is updated (if relevant)
    • Commit sequence broadly makes sense
    • Key commits have useful messages
  • PR
    • No clippy warnings in the CI
    • Self-reviewed the diff
    • Useful pull request description
    • Reviewer requested

Issue(s)

Closes #2021

@Alenar Alenar self-assigned this Oct 21, 2024
Copy link

github-actions bot commented Oct 21, 2024

Test Results

    4 files  ± 0     55 suites  ±0   10m 41s ⏱️ +19s
1 393 tests +14  1 393 ✅ +14  0 💤 ±0  0 ❌ ±0 
1 607 runs  +14  1 607 ✅ +14  0 💤 ±0  0 ❌ ±0 

Results for commit 84ca63f. ± Comparison against base commit af96205.

♻️ This comment has been updated with latest results.

Copy link
Member

@jpraynaud jpraynaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@Alenar Alenar force-pushed the djo/2021/tx-chunk-importer_avoid-uneeded-rescan branch 3 times, most recently from e0a921a to b513d9a Compare October 22, 2024 10:30
@Alenar Alenar force-pushed the djo/2021/tx-chunk-importer_avoid-uneeded-rescan branch from b513d9a to 1b6e982 Compare October 22, 2024 12:48
…ackward`

This will allow to return the latest chain point known by the cardano
block scanner.
Else we only know it when there's a roll foward.
…chunk

This fix the issue of importing when there's no transactions to store in
a large range of blocks:
* Instead of always using the database to get the start point the db is
  only used for the first chunk.
* For subsequent call the last chain point polled by the streamer is
  used.
@Alenar Alenar force-pushed the djo/2021/tx-chunk-importer_avoid-uneeded-rescan branch from 1b6e982 to 5f04634 Compare October 22, 2024 14:06
* mithril-aggregator from `0.5.87` to `0.5.88`
* mithril-common from `0.4.72` to `0.4.73`
* mithril-signer from `0.2.202` to `0.2.203`
@Alenar Alenar force-pushed the djo/2021/tx-chunk-importer_avoid-uneeded-rescan branch from 5f04634 to 84ca63f Compare October 22, 2024 14:47
@Alenar Alenar merged commit d36e448 into main Oct 22, 2024
45 of 53 checks passed
@Alenar Alenar deleted the djo/2021/tx-chunk-importer_avoid-uneeded-rescan branch October 22, 2024 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix Cardano transaction chunk importer
4 participants