Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: dont' persist storage changes at genesis #1182

Merged
merged 10 commits into from
Mar 13, 2023

Conversation

batconjurer
Copy link
Member

@batconjurer batconjurer commented Feb 23, 2023

Persisting storage at genesis (init chain) can cause database corruption if a node restarts before reaching blockheight 1. The persistence of state should only occur at the time that tendermint is informed of the next block height. Thus we remove the db write call in init chain and defer persisting until the first commit block call.

Note: Due to Issue #1190, not all e2e tests may pass.

@grarco
Copy link
Contributor

grarco commented Feb 23, 2023

Should we also change init_chain to accept an immutable &self reference?

@Fraccaman Fraccaman changed the title [feat]: Dont' persist storage changes at genesis fix: dont' persist storage changes at genesis Feb 23, 2023
@batconjurer batconjurer marked this pull request as draft February 23, 2023 16:39
@batconjurer batconjurer requested review from tzemanovic and removed request for Fraccaman and brentstone February 24, 2023 08:19
@batconjurer
Copy link
Member Author

I'm not sure why this fixes the db corruption. There are other storage writes in init_chain that seems like they would be persisted if the ledger were shut down prior to the first commit_block. I'm basing this on the fact that flush is called on rocksdb on drop. However, I don't observe this locally.

@batconjurer batconjurer marked this pull request as ready for review February 24, 2023 08:21
@batconjurer
Copy link
Member Author

I'm not sure why this fixes the db corruption. There are other storage writes in init_chain that seems like they would be persisted if the ledger were shut down prior to the first commit_block. I'm basing this on the fact that flush is called on rocksdb on drop. However, I don't observe this locally.

Ok, I thing I've resolved my confusion. The point is that we are no longer committing a block, so the flushed data in the sst files gets ignored.

@tzemanovic
Copy link
Member

I'm not sure why this fixes the db corruption. There are other storage writes in init_chain that seems like they would be persisted if the ledger were shut down prior to the first commit_block. I'm basing this on the fact that flush is called on rocksdb on drop. However, I don't observe this locally.

Ok, I thing I've resolved my confusion. The point is that we are no longer committing a block, so the flushed data in the sst files gets ignored.

The stuff that's written into wl_storage goes into write log at block level, so nothing goes into storage until it's committed

@tzemanovic
Copy link
Member

Looks like bunch e2e tests are affected?

@batconjurer batconjurer marked this pull request as draft March 1, 2023 09:37
@tzemanovic tzemanovic force-pushed the bat/hotfix/dont-persist-gensis-at-init-chain branch from 9054bb4 to 9f12e09 Compare March 2, 2023 09:54
@batconjurer batconjurer marked this pull request as ready for review March 2, 2023 13:42
@tzemanovic tzemanovic force-pushed the bat/hotfix/dont-persist-gensis-at-init-chain branch 3 times, most recently from f365980 to cc27ab0 Compare March 6, 2023 07:15
@tzemanovic
Copy link
Member

Should we also change init_chain to accept an immutable &self reference?

it does have to be mutable as it has to write to write log and to initialize couple things in storage in-memory, but it should not commit to DB - I've added a unit test in 8d8b563 to check that - there were couple things doing this (chain parameters, gov params and IBC)

@tzemanovic tzemanovic force-pushed the bat/hotfix/dont-persist-gensis-at-init-chain branch from cc27ab0 to 1694af8 Compare March 6, 2023 08:47
@tzemanovic tzemanovic force-pushed the bat/hotfix/dont-persist-gensis-at-init-chain branch from 1694af8 to f053725 Compare March 6, 2023 09:52
@tzemanovic tzemanovic force-pushed the bat/hotfix/dont-persist-gensis-at-init-chain branch from 5d3b9e1 to 979a5ec Compare March 6, 2023 12:02
apps/src/lib/node/ledger/shell/init_chain.rs Outdated Show resolved Hide resolved
@tzemanovic
Copy link
Member

needs more reviews

juped added a commit that referenced this pull request Mar 13, 2023
* tag 'v0.14.2':
  Namada 0.14.2
  changelog: add #1191
  test/pos/sm: fix init-validator and bond pre-conditions
  [ci] wasm checksums update
  ci: use nightly version for e2e test
  test/pos/sm: add the rest of the conditions
  pos: improve withdrawal logs
  test/pos/sm: add another bonds post-cond
  test/pos/sm: generate InitValidator transitions
  test/pos: fix the bonds test
  test/pos: reduce the bond token amounts to cover cases with same amounts
  bug fix: `update_validator_set` precisely checks if validator in consensus set
  pos: remove the `init` function to just use `set` instead
  make: add unstable-options to `check-abcipp` recipe
  pos: turn prints into tracing::debug, tidy up code
  pos/epoched: fix the update_data logic
  test/pos: add a state machine test
  test/core/address: fix address generator to be deterministic
  core/token: re-export `token::Change` type from storage_api mod
  make: use unstable-options to build unit tests
  changelog: add #1197
  [ci] wasm checksums update
  pos: ensure that validator consensus keys are unique
  core/storage: impl KeySeg for common::PublicKey
  test/lazy_set: add `try_insert` to state machine test
  core/lazy_set: add `try_insert` method
  small documentation edits
  changelog: add #1196
  [ci] wasm checksums update
  test: add a state machine test for lazy set collection
  core/storage_api: add LazySet
  changelog: #1182
  test/e2e: wait for a first block before client cmds
  wl_storage: remove commit_genesis method
  test/e2e: put ledger to bg to avoid it getting stuck
  gov/parameters: init via storage_api write log
  parameters: init chain parameters via storage_api write log
  init-chain: fix ibc to go via wl_storage
  [chore]:Added a doc warning
  [feat]: Dont' persist storage changes at genesis
  test/init_chain: ensure that init-chain doesn't commit to DB
  test/storage: reduce arb key length
  [ci] wasm checksums update
  changelog: add #1141
  bug fix: reliable deterministic ordering of keys in wl_storage PrefixIter that fixes apply_inflation bug
  test/core/wl_storage: add test for `prefix_iter_pre`/`prefix_iter_post`
@juped juped merged commit 1da46fb into main Mar 13, 2023
@juped juped deleted the bat/hotfix/dont-persist-gensis-at-init-chain branch March 13, 2023 21:34
bengtlofgren pushed a commit that referenced this pull request May 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants