feat(prune, stages): prune pipeline stage #9419

shekhirin · 2024-07-10T14:52:13Z

reth/crates/stages/stages/src/stages/prune.rs

Lines 9 to 16 in 232c6ae

    
           /// The prune stage that runs the pruner with the provided prune modes. 
        
           /// 
        
           /// There are two main reasons to have this stage when running a full node: 
        
           /// - Sender Recovery stage inserts a lot of data into the database that's only needed for the 
        
           ///   Execution stage. Pruner will clean up the unneeded recovered senders. 
        
           /// - Pruning during the live sync can take a significant amount of time, especially history 
        
           ///   segments. If we can prune as much data as possible in one go before starting the live sync, we 
        
           ///   should do it.

crates/prune/prune/src/builder.rs

crates/prune/prune/src/pruner.rs

crates/stages/stages/src/sets.rs

mattsse

some questions.

will this now always log on CLI output even if archive node?
should this be optionally installed if prunemodes none?

crates/prune/prune/src/pruner.rs

crates/stages/api/src/pipeline/mod.rs

crates/stages/stages/src/stages/prune.rs

crates/stages/types/src/id.rs

crates/prune/prune/src/pruner.rs

mattsse

last nits

crates/prune/prune/src/pruner.rs

crates/stages/stages/src/sets.rs

mattsse

cool, last naming nits

crates/stages/api/src/pipeline/set.rs

crates/stages/stages/src/sets.rs

feat(prune, stages): prune pipeline stage

96ce62b

shekhirin added C-enhancement New feature or request A-staged-sync Related to staged sync (pipelines and stages) A-pruning Related to pruning or full node labels Jul 10, 2024

shekhirin added 2 commits July 10, 2024 16:04

fix some lints

65730f3

return previous checkpoint if pruner didn't finish

47669ba

shekhirin force-pushed the alexey/prune-stage branch from a8be3f3 to 47669ba Compare July 10, 2024 16:18

add test

232c6ae

shekhirin marked this pull request as ready for review July 10, 2024 17:07

shekhirin requested review from onbjerg, rkrasiuk, joshieDo, mattsse, Rjected, fgimenez and gakonst as code owners July 10, 2024 17:07

shekhirin requested a review from emhane July 10, 2024 17:07

emhane requested changes Jul 11, 2024

View reviewed changes

crates/prune/prune/src/builder.rs Show resolved Hide resolved

crates/prune/prune/src/builder.rs Show resolved Hide resolved

crates/prune/prune/src/pruner.rs Outdated Show resolved Hide resolved

crates/stages/stages/src/sets.rs Outdated Show resolved Hide resolved

mattsse reviewed Jul 11, 2024

View reviewed changes

crates/prune/prune/src/pruner.rs Show resolved Hide resolved

crates/stages/api/src/pipeline/mod.rs Outdated Show resolved Hide resolved

crates/stages/stages/src/stages/prune.rs Show resolved Hide resolved

crates/stages/types/src/id.rs Show resolved Hide resolved

mattsse requested changes Jul 11, 2024

View reviewed changes

crates/prune/prune/src/pruner.rs Show resolved Hide resolved

shekhirin added 3 commits July 11, 2024 14:08

replace states with just one generic arg

e460160

Merge remote-tracking branch 'origin/main' into alexey/prune-stage

3b37bee

only add prune stage if any prune modes are set

b82b312

shekhirin force-pushed the alexey/prune-stage branch from c9e2c3e to b82b312 Compare July 11, 2024 13:36

make prune stage last

207aeeb

mattsse requested changes Jul 11, 2024

View reviewed changes

crates/prune/prune/src/pruner.rs Outdated Show resolved Hide resolved

crates/stages/stages/src/sets.rs Outdated Show resolved Hide resolved

shekhirin added 3 commits July 11, 2024 15:54

add a separate prune sender recovery stage

4c42a75

flip the generics

8c06f88

add add_stage_if

2746051

shekhirin requested a review from mattsse July 11, 2024 15:15

shekhirin requested a review from emhane July 11, 2024 15:15

mattsse approved these changes Jul 11, 2024

View reviewed changes

crates/stages/api/src/pipeline/set.rs Outdated Show resolved Hide resolved

crates/stages/stages/src/sets.rs Outdated Show resolved Hide resolved

style by matt

42787f3

emhane approved these changes Jul 11, 2024

View reviewed changes

shekhirin added this pull request to the merge queue Jul 11, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jul 11, 2024

shekhirin added 2 commits July 11, 2024 19:58

Merge remote-tracking branch 'origin/main' into alexey/prune-stage

c311d25

fix persistence handle test

218ef7a

shekhirin enabled auto-merge July 11, 2024 19:03

shekhirin added this pull request to the merge queue Jul 11, 2024

Merged via the queue into main with commit b040b86 Jul 11, 2024
33 checks passed

shekhirin deleted the alexey/prune-stage branch July 11, 2024 19:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(prune, stages): prune pipeline stage #9419

feat(prune, stages): prune pipeline stage #9419

shekhirin commented Jul 10, 2024 •

edited

Loading

mattsse left a comment

mattsse left a comment

mattsse left a comment

	/// The prune stage that runs the pruner with the provided prune modes.
	///
	/// There are two main reasons to have this stage when running a full node:
	/// - Sender Recovery stage inserts a lot of data into the database that's only needed for the
	/// Execution stage. Pruner will clean up the unneeded recovered senders.
	/// - Pruning during the live sync can take a significant amount of time, especially history
	/// segments. If we can prune as much data as possible in one go before starting the live sync, we
	/// should do it.

feat(prune, stages): prune pipeline stage #9419

feat(prune, stages): prune pipeline stage #9419

Conversation

shekhirin commented Jul 10, 2024 • edited Loading

mattsse left a comment

Choose a reason for hiding this comment

mattsse left a comment

Choose a reason for hiding this comment

mattsse left a comment

Choose a reason for hiding this comment

shekhirin commented Jul 10, 2024 •

edited

Loading