Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save BlockView after non_contextual_verify, Remove ConsumeOrphan thread, Preload UnverifiedBlock for ConsumeUnverified thread. #4391

Conversation

eval-exec
Copy link
Collaborator

@eval-exec eval-exec commented Mar 26, 2024

What problem does this PR solve?

Want to merge into #4365

This PR mainly aims to have the ChainService store the block into the db as soon as it receives it and completes the non_contextual_verify. This way, the OrphanBroker only saves the smaller-sized LonelyBlockHash in memory. Additionally, a PreloadUnverified thread is added to preload UnverifiedBlocks from the db, which are then consumed by the ConsumeUnverified thread.

The advantage of this approach is that even when syncing large blocks during moments like the Omiga, the memory usage of CKB will not significantly increase.

What's Changed:

Related changes

  • Save BlockView to db after non_contextual_verify
  • Remove ConsumeOrphan thread, let OrphanBroker only record LonelyBlockHash in memory
  • Created a thread: PreloadUnverifiedBlock to pre-load UnverifiedBlock for ConsumeUnverified thread.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code ci-runs-only: [ quick_checks,linters ]

Side effects

  • None

Release note

None: Exclude this PR from the release note.

@eval-exec eval-exec requested a review from a team as a code owner March 26, 2024 09:27
@eval-exec eval-exec requested review from doitian and removed request for a team March 26, 2024 09:27
@eval-exec eval-exec changed the title Save BlockView after non_contextual_verify, Remove ConsumeOrphan thread, Preload unverified blocks for ConsumeUnverified thread. Save BlockView after non_contextual_verify, Remove ConsumeOrphan thread, Preload UnverifiedBlock for ConsumeUnverified thread. Mar 26, 2024
@eval-exec eval-exec force-pushed the exec/ckb-async-download/remove-orphan-pool branch from f946c39 to d98bf1c Compare March 26, 2024 09:33
@eval-exec eval-exec marked this pull request as draft March 26, 2024 09:35
@eval-exec eval-exec force-pushed the exec/ckb-async-download/remove-orphan-pool branch from d98bf1c to 8ef1d95 Compare March 26, 2024 09:51
@eval-exec eval-exec force-pushed the exec/ckb-async-download/remove-orphan-pool branch 3 times, most recently from b267cd0 to 40b203c Compare March 26, 2024 17:00
@eval-exec
Copy link
Collaborator Author

image
From genesis block to 12,500,000:
v0.115.0-rc2: need 30h40min
this PR: need 23h

This PR reduces the synchronization time by approximately 25%.

@eval-exec
Copy link
Collaborator Author

eval-exec commented Mar 26, 2024

Compare v0.115.0-rc2 vs this PR's full synchronization RSS mem usage:
image

only compare Omiga huge blocks:
image

In 11,980,000 -> 11,995,000:
v0.115.0-rc2: RSS mem usage fluctuates between 1.9G and 2.5G.
With this PR, the RSS memory usage consistently stays at 2.0G.

@eval-exec eval-exec marked this pull request as ready for review March 26, 2024 17:20
…o pre load unverified block

Signed-off-by: Eval EXEC <execvy@gmail.com>
@eval-exec eval-exec force-pushed the exec/ckb-async-download/remove-orphan-pool branch from 40b203c to d7b85ca Compare March 27, 2024 02:34
@eval-exec
Copy link
Collaborator Author

I believe that merging this PR into #4365 will make the review process more convenient.

@eval-exec eval-exec merged commit fe63750 into nervosnetwork:ckb-async-download Mar 27, 2024
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant