core/state/snapshot, ethdb: track deletions more accurately #22582
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // Ensure we don't write too much data blindly. It's ok to flush, the | ||
| // root will go missing in case of a crash and we'll detect and regen | ||
| // the snapshot. | ||
| if batch.ValueSize() > ethdb.IdealBatchSize { |
There was a problem hiding this comment.
Shouldn't we also flush the "full" batch when we try to write the storages?
There was a problem hiding this comment.
The important one is the delete-flush, since it may be unbounded (extremely large). The things we hold in memory are bounded by the memory limits of the difflayers.
karalabe
force-pushed
the
snapshot-delflat-fix
branch
from
c113a5a
to
e739b02
Compare
karalabe
force-pushed
the
snapshot-delflat-fix
branch
from
e739b02
to
7977600
Compare
atif-konasl
pushed a commit
to frozeman/pandora-execution-engine
that referenced
this pull request
Oct 15, 2021
…#22582) * core/state/snapshot, ethdb: track deletions more accurately * core/state/snapshot: don't reset the iterator, leveldb's screwy * ethdb: don't mess with the insert batches for now
tony-ricciardi
pushed a commit
to tony-ricciardi/go-ethereum
that referenced
this pull request
Jan 20, 2022
Cherry pick bug fixes from upstream for snapshots, which will enable higher transaction throughput. It also enables snapshots by default (which is one of the commits pulled from upstream). Upstream commits included: 68754f3 cmd/utils: grant snapshot cache to trie if disabled (ethereum#21416) 3ee91b9 core/state/snapshot: reduce disk layer depth during generation a15d71a core/state/snapshot: stop generator if it hits missing trie nodes (ethereum#21649) 43c278c core/state: disable snapshot iteration if it's not fully constructed (ethereum#21682) b63e3c3 core: improve snapshot journal recovery (ethereum#21594) e640267 core/state/snapshot: fix journal recovery from generating old journal (ethereum#21775) 7b7b327 core/state/snapshot: update generator marker in sync with flushes 167ff56 core/state/snapshot: gethring -> gathering typo (ethereum#22104) d2e1b17 snapshot, trie: fixed typos, mostly in snapshot pkg (ethereum#22133) c4deebb core/state/snapshot: add generation logs to storage too 5e9f5ca core/state/snapshot: write snapshot generator in batch (ethereum#22163) 18145ad core/state: maintain one more diff layer (ethereum#21730) 04a7226 snapshot: merge loops for better performance (ethereum#22160) 994cdc6 cmd/utils: enable snapshots by default 9ec3329 core/state/snapshot: ensure Cap retains a min number of layers 52e5c38 core/state: copy the snap when copying the state (ethereum#22340) a31f6d5 core/state/snapshot: fix panic on missing parent 61ff3e8 core/state/snapshot, ethdb: track deletions more accurately (ethereum#22582) c79fc20 core/state/snapshot: fix data race in diff layer (ethereum#22540) Other changes Commit f9b5530 (not from upstream) fixes an incorrect default DatabaseCache value due to an earlier bad merge. Tested Automated tests Testing on a private testnet Backwards compatibility Enabling snapshots by default is a breaking change in terms of the CLI flags, but will not cause backwards incompatibility between the node and other nodes. Co-authored-by: Péter Szilágyi <peterke@gmail.com> Co-authored-by: gary rong <garyrong0905@gmail.com> Co-authored-by: Melvin Junhee Woo <melvin.woo@groundx.xyz> Co-authored-by: Martin Holst Swende <martin@swende.se> Co-authored-by: Edgar Aroutiounian <edgar.factorial@gmail.com>
9 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We've seen some high memory usage on Ropsten around block 6M when snapshots are enabled. Our hunch is that there might be some huge contract self-destruct that takes it's toll on the memory usage. This PR is a benchmark test to see if that's indeed the case or not.