Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(pruner): respect batch size per run #4246

Merged
merged 25 commits into from
Aug 23, 2023
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
cf670f2
feat(pruner): respect batch size per run
shekhirin Aug 17, 2023
689d267
new batch logic
shekhirin Aug 18, 2023
d9115be
fix engine prune event
shekhirin Aug 18, 2023
bbf5c0c
more trace logs
shekhirin Aug 18, 2023
adffcf6
even more logs
shekhirin Aug 18, 2023
8fa4757
fix short-circuit
shekhirin Aug 18, 2023
80a53ef
improve log fields
shekhirin Aug 18, 2023
1d149e0
increase batch sizes to 1000
shekhirin Aug 18, 2023
54a67f6
add comments
shekhirin Aug 18, 2023
4939ab9
Merge remote-tracking branch 'origin/main' into alexey/pruner-batch-size
shekhirin Aug 21, 2023
8451bed
fix stage checkpoint tests
shekhirin Aug 21, 2023
c10eeb5
fixes of receipts by logs pruner checkpoint
shekhirin Aug 21, 2023
df14e1e
return early if receipts by logs block range is empty
shekhirin Aug 22, 2023
0498b18
Revert "return early if receipts by logs block range is empty"
shekhirin Aug 22, 2023
a092d72
check empty tx range
shekhirin Aug 22, 2023
bedb5a4
check tip block number
shekhirin Aug 22, 2023
1ac798b
Merge remote-tracking branch 'origin/main' into alexey/pruner-batch-size
shekhirin Aug 22, 2023
08f71e2
add another log for contracts by logs pruning
shekhirin Aug 22, 2023
b941ed4
prune receipts by logs with range instead of iterator
shekhirin Aug 22, 2023
9b4925a
Merge remote-tracking branch 'origin/main' into alexey/pruner-batch-size
shekhirin Aug 22, 2023
de2d739
Merge remote-tracking branch 'origin/main' into alexey/pruner-batch-size
shekhirin Aug 22, 2023
d4e760a
fix checkpoints for contract log filtering on live sync
joshieDo Aug 22, 2023
a042bf6
fix lint
shekhirin Aug 23, 2023
7d43486
fixes after review
shekhirin Aug 23, 2023
407bdcc
Merge remote-tracking branch 'origin/main' into alexey/pruner-batch-size
shekhirin Aug 23, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 18 additions & 37 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions crates/consensus/beacon/src/engine/prune.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
use futures::FutureExt;
use reth_db::database::Database;
use reth_primitives::BlockNumber;
use reth_prune::{Pruner, PrunerError, PrunerWithResult};
use reth_prune::{Pruner, PrunerResult, PrunerWithResult};
use reth_tasks::TaskSpawner;
use std::task::{ready, Context, Poll};
use tokio::sync::oneshot;
Expand Down Expand Up @@ -116,7 +116,7 @@ pub(crate) enum EnginePruneEvent {
/// If this is returned, the pruner is idle.
Finished {
/// Final result of the pruner run.
result: Result<(), PrunerError>,
result: PrunerResult,
},
/// Pruner task was dropped after it was started, unable to receive it because channel
/// closed. This would indicate a panicked pruner task
Expand Down
5 changes: 4 additions & 1 deletion crates/interfaces/src/test_utils/generators.rs
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,7 @@ where
// deposit in receiving account and update storage
let (prev_to, storage): &mut (Account, BTreeMap<H256, U256>) = state.get_mut(&to).unwrap();

let old_entries = new_entries
let mut old_entries: Vec<_> = new_entries
.into_iter()
.filter_map(|entry| {
let old = if entry.value != U256::ZERO {
Expand All @@ -254,9 +254,12 @@ where
Some(StorageEntry { value: old.unwrap_or(U256::from(0)), ..entry })
})
.collect();
old_entries.sort_by_key(|entry| entry.key);

changeset.push((to, *prev_to, old_entries));

changeset.sort_by_key(|(address, _, _)| *address);

prev_to.balance = prev_to.balance.wrapping_add(transfer);

changesets.push(changeset);
Expand Down
7 changes: 5 additions & 2 deletions crates/primitives/src/prune/checkpoint.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use crate::{prune::PruneMode, BlockNumber};
use crate::{prune::PruneMode, BlockNumber, TxNumber};
use reth_codecs::{main_codec, Compact};

/// Saves the pruning progress of a stage.
Expand All @@ -7,7 +7,10 @@ use reth_codecs::{main_codec, Compact};
#[cfg_attr(test, derive(Default))]
pub struct PruneCheckpoint {
/// Highest pruned block number.
pub block_number: BlockNumber,
/// If it's [None], the pruning for block `0` is not finished yet.
pub block_number: Option<BlockNumber>,
/// Highest pruned transaction number, if applicable.
pub tx_number: Option<TxNumber>,
/// Prune mode.
pub prune_mode: PruneMode,
}
6 changes: 5 additions & 1 deletion crates/primitives/src/prune/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -49,10 +49,14 @@ impl ContractLogsPruneConfig {
// the BTreeMap (block = 0), otherwise it will be excluded.
// Reminder that this BTreeMap works as an inclusion list that excludes (prunes) all
// other receipts.
//
// Reminder, that we increment because the [`BlockNumber`] key of the new map should be
// viewed as `PruneMode::Before(block)`
let block = (pruned_block + 1).max(
mode.prune_target_block(tip, MINIMUM_PRUNING_DISTANCE, PrunePart::ContractLogs)?
.map(|(block, _)| block)
.unwrap_or_default(),
.unwrap_or_default() +
1,
);

map.entry(block).or_insert_with(Vec::new).push(address)
Expand Down
Loading
Loading