Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[R4R]db: freezer batch compatible offline prunblock command #1005

Merged
merged 5 commits into from
Jul 22, 2022
Merged

[R4R]db: freezer batch compatible offline prunblock command #1005

merged 5 commits into from
Jul 22, 2022

Conversation

joeylichang
Copy link
Contributor

@joeylichang joeylichang commented Jul 20, 2022 •
edited
Loading

Description

  • Freezer batch compatible offline prune tool.
  • Normalized the prune-block command parameters to avoid the first execution failure for new users due to parameter problems.

Changes

Usage

  • geth snapshot prune-block --datadir $datadir_var --datadir.ancient $ancientdir_var
    | --datadir and --datadir.ancient parameters are required options and uses absolute path

Code changes:

  • add offset(after offline prune tool running, offset stands the start block number in ancient) to freezer batch.
  • batch reset and commit will consider offset.

flywukong
flywukong reviewed Jul 20, 2022
View reviewed changes
cmd/geth/snapshot.go Outdated Show resolved Hide resolved
@unclezoro unclezoro changed the title freezer batch compatible offline prunblock command [R4R]db: freezer batch compatible offline prunblock command Jul 20, 2022
Mercybudda
Mercybudda reviewed Jul 20, 2022
View reviewed changes
core/rawdb/database.go Show resolved Hide resolved
core/state/pruner/pruner.go Outdated Show resolved Hide resolved
richardrich975
richardrich975 previously approved these changes Jul 20, 2022
View reviewed changes
Copy link

@richardrich975 richardrich975 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

core/state/pruner/pruner.go Outdated Show resolved Hide resolved
cmd/geth/snapshot.go Outdated Show resolved Hide resolved
Mercybudda
Mercybudda previously approved these changes Jul 20, 2022
View reviewed changes
Copy link
Contributor

@Mercybudda Mercybudda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please pass all UTs check then.

flywukong
flywukong previously approved these changes Jul 20, 2022
View reviewed changes
richardrich975
richardrich975 approved these changes Jul 22, 2022
View reviewed changes
Copy link

@richardrich975 richardrich975 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@unclezoro unclezoro merged commit d93211b into develop Jul 22, 2022
@joeylichang joeylichang deleted the offline_prune_compatible_freezer_batch branch July 23, 2022 15:30
This was referenced Jul 28, 2022
Merged
Merged
jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023
@jsvisa
core/rawdb: freezer batch should implement the offset commit, ref bnb…
1520cb5 
…-chain/bsc#1005

Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023
@jsvisa
core/rawdb: freezer batch should implement the offset commit, ref bnb…
ecffeda 
…-chain/bsc#1005

Signed-off-by: Delweng <delweng@gmail.com>
@jsvisa jsvisa mentioned this pull request Feb 21, 2023
Closed
16 tasks
jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 1, 2023
@jsvisa
core/rawdb: freezer batch should implement the offset commit, ref bnb…
1e2cb93 
…-chain/bsc#1005

Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 17, 2023
@jsvisa
core/rawdb: freezer batch should implement the offset commit, ref bnb…
f7833b7 
…-chain/bsc#1005

Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Jun 28, 2023
@jsvisa
core/rawdb: freezer batch should implement the offset commit, ref bnb…
8a44857 
…-chain/bsc#1005

Signed-off-by: Delweng <delweng@gmail.com>
manav2401 pushed a commit to maticnetwork/bor that referenced this pull request Apr 9, 2024
@jsvisa @manav2401
core/rawdb: freezer batch should implement the offset commit, ref bnb…
cf08b90 
…-chain/bsc#1005

Signed-off-by: Delweng <delweng@gmail.com>
manav2401 added a commit to maticnetwork/bor that referenced this pull request May 9, 2024
@manav2401 @jsvisa
Enable ancient block pruning (#1216)
d95c05b 
* core/state: typo

Signed-off-by: Delweng <delweng@gmail.com>

* core/rawdb: backport from bnb-chain/bsc#543

Signed-off-by: Delweng <delweng@gmail.com>

* eth,ethdb,node,core/state: backport from bnb-chain/bsc#543

Signed-off-by: Delweng <delweng@gmail.com>

* eth,core: backport from bnb-chain/bsc#543

Signed-off-by: Delweng <delweng@gmail.com>

* cmd: open db with freeze disabled

Signed-off-by: Delweng <delweng@gmail.com>

* cli: snapshot prune-block

Signed-off-by: Delweng <delweng@gmail.com>

* fix typo

Signed-off-by: Delweng <delweng@gmail.com>

* cli/snapshot: fix the issue of dup open db error

Signed-off-by: Delweng <delweng@gmail.com>

* cli/snapshot: resolve datadir and ancient before backup

Signed-off-by: Delweng <delweng@gmail.com>

* core: more prune-block log

Signed-off-by: Delweng <delweng@gmail.com>

* core: truncatetail missing f.offset

Signed-off-by: Delweng <delweng@gmail.com>

* core/rawdb: indextx adjust offset of pruned block

Signed-off-by: Delweng <delweng@gmail.com>

* core/rawdb: freezer batch should implement the offset commit, ref bnb-chain/bsc#1005

Signed-off-by: Delweng <delweng@gmail.com>

* core: check of ancientdb, backport bnb-chain/bsc#817

Signed-off-by: Delweng <delweng@gmail.com>

* core/state: read raw borReceipt to backup

Signed-off-by: Delweng <delweng@gmail.com>

* core/rawdb: bor receipt maybe in []Receipt or Receipt RLP format

Signed-off-by: Delweng <delweng@gmail.com>

* core/state: typo and error msg

Signed-off-by: Delweng <delweng@gmail.com>

* core/rawdb: offSet -> offset

Signed-off-by: Delweng <delweng@gmail.com>

* cli/snapshot: comment

Signed-off-by: Delweng <delweng@gmail.com>

* cli/snapshot: add prune-block doc

Signed-off-by: Delweng <delweng@gmail.com>

* docs: add prune-block document

Signed-off-by: Delweng <delweng@gmail.com>

* core/rawdb: print wrong bor-receipt length

Signed-off-by: Delweng <delweng@gmail.com>

* internal/cli: add snapshot prune block tests (referenced from bsc's PR)

* improve errors

* cmd, core, eth, internal: fix lint

* internal/cli: refactor snapshot prune block test

* fix linters in tests

* internal/cli: add inspect-ancient-db command, update docs

* pruner: use a generic function for simplification

* internal/cli: fixes for inspect-db command

* internal/cli: improve pruning tests

* core/rawdb: update end block calculation logic in inspect command

* core/rawdb: improve checks db initialisation

* core/rawdb: remove offset check

* update mocks for span, ethdb and add command in makefile

* docs/cli: update docs with inspect command

* go mod tidy

* refactor and resolve conflicts

* resolve more conflicts

* refactor

* explicitly read node for hash scheme

* add check for hash scheme, fix tests

* fix typo

* update docs and add warning

* raise error if pbss is enabled

* revert read raw bor receipt change

* consensus/bor: handle nil header case in get root hash

* address comments

* core/rawdb: check chain continuity by matching parent hash

* core/rawdb: account for pruned ancient blocks

* go mod tidy

* fix tests

* fix tests

---------

Signed-off-by: Delweng <delweng@gmail.com>
Co-authored-by: Delweng <delweng@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@richardrich975 richardrich975 richardrich975 approved these changes

@calmbeing calmbeing calmbeing approved these changes

@Mercybudda Mercybudda Mercybudda approved these changes

@unclezoro unclezoro unclezoro approved these changes

@flywukong flywukong flywukong left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@joeylichang @unclezoro @flywukong @Mercybudda @calmbeing @richardrich975