Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem: production rocksdb configuration is not optimal #813

Merged
merged 11 commits into from
Jan 19, 2023

Conversation

yihuang
Copy link
Collaborator

@yihuang yihuang commented Jan 18, 2023

Solution:

During the versiondb migration experiments, I found the default rocksdb settings are multiple times slower than the one I experimented, because I've tuned options and did a full compaction on the db before, but after I keep the node running with the normal binary for a while, suddenly my script's performance drop significantly, and the sst files are all rewritten by compaction process.
I suspect the main difference is sst file sizes, and probably filter and compression types as well, so I plan to tune the rocksdb options for default binary first, then try the versiondb migration script again, to see if it can get back to the old speed.

Compare Compression Options

  • Tool: sst_dump --file=sample.sst --command=recompress ...
  • Block size: 32k
Compression Type Size Time
Snappy 228861267 1.71s
LZ4 228341872 1.85s
ZSTD 217266835 3.93
ZSTD level 11 214701135 13.7s
ZSTD level 11 with 11k dictionary 186643202 20.6s
ZSTD level 11 with 110k dictionary 177305609 22.4s

Basically I picked relatively slower speed and higher compression ratio for the bottommost level, I think our node do have the capacity to occupy one core to do the heavier compaction gradually.

👮🏻👮🏻👮🏻 !!!! REFERENCE THE PROBLEM YOUR ARE SOLVING IN THE PR TITLE AND DESCRIBE YOUR SOLUTION HERE !!!! DO NOT FORGET !!!! 👮🏻👮🏻👮🏻

PR Checklist:

  • Have you read the CONTRIBUTING.md?
  • Does your PR follow the C4 patch requirements?
  • Have you rebased your work on top of the latest master?
  • Have you checked your code compiles? (make)
  • Have you included tests for any non-trivial functionality?
  • Have you checked your code passes the unit tests? (make test)
  • Have you checked your code formatting is correct? (go fmt)
  • Have you checked your basic code style is fine? (golangci-lint run)
  • If you added any dependencies, have you checked they do not contain any known vulnerabilities? (go list -json -m all | nancy sleuth)
  • If your changes affect the client infrastructure, have you run the integration test?
  • If your changes affect public APIs, does your PR follow the C4 evolution of public contracts?
  • If your code changes public APIs, have you incremented the crate version numbers and documented your changes in the CHANGELOG.md?
  • If you are contributing for the first time, please read the agreement in CONTRIBUTING.md now and add a comment to this pull request stating that your PR is in accordance with the Developer's Certificate of Origin.

Thank you for your code, it's appreciated! :)

Solution:
- update related dependencies to allow customize rocksdb options.
- especially using rocksdb v7.
- tune rocksdb options.
Makefile Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
@yihuang yihuang marked this pull request as ready for review January 18, 2023 16:32
@yihuang yihuang requested a review from a team as a code owner January 18, 2023 16:32
@yihuang yihuang requested review from mmsqe, thomas-nguy, JayT106 and a team and removed request for a team January 18, 2023 16:32
Copy link
Collaborator

@mmsqe mmsqe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still get could not determine kind of name for C.rocksdb_lru_cache_options_set_num_shard_bits build error in grocksdb

@yihuang
Copy link
Collaborator Author

yihuang commented Jan 19, 2023

still get could not determine kind of name for C.rocksdb_lru_cache_options_set_num_shard_bits build error in grocksdb

try recent version of rocksdb, or using nix, the nix script use v7.8.3.

@yihuang
Copy link
Collaborator Author

yihuang commented Jan 19, 2023

@JayT106 I'll merge now to unblock some other fixes, feel free to further discuss the options picked here.

@yihuang yihuang enabled auto-merge (squash) January 19, 2023 02:58
@yihuang yihuang merged commit 7ef5b04 into crypto-org-chain:release/v1.0.x Jan 19, 2023
@yihuang yihuang deleted the tune-rocksdb-options branch January 19, 2023 03:19
@JayT106
Copy link
Collaborator

JayT106 commented Jan 19, 2023

file descriptors should not be a problem, if ram usage is a concern we should enable SetCacheIndexAndFilterBlocks, and with bigger sst file size (SetTargetFileSizeMultiplier) and block size (32k), the total amount of filter and index should be reduced.

I remembered the value SetMaxOpenFiles was affecting the RAM usage a lot when exporting the genesis file. >64GB using when the value is 4096 and <32GB using when it is 1000
Maybe we need to evaluate it against the new settings. Before the genesis export feature is settled in the SDK, it will always be an issue.

@yihuang
Copy link
Collaborator Author

yihuang commented Jan 20, 2023

file descriptors should not be a problem, if ram usage is a concern we should enable SetCacheIndexAndFilterBlocks, and with bigger sst file size (SetTargetFileSizeMultiplier) and block size (32k), the total amount of filter and index should be reduced.

I remembered the value SetMaxOpenFiles was affecting the RAM usage a lot when exporting the genesis file. >64GB using when the value is 4096 and <32GB using when it is 1000 Maybe we need to evaluate it against the new settings. Before the genesis export feature is settled in the SDK, it will always be an issue.

can you try SetCacheIndexAndFilterBlocks(true) to control ram usage during genesis export?
I think the reason open file limit affect ram usage is because the index/filter caches, CacheIndexAndFilterBlocks will manage those caches together with block cache.

@JayT106
Copy link
Collaborator

JayT106 commented Jan 25, 2023

file descriptors should not be a problem, if ram usage is a concern we should enable SetCacheIndexAndFilterBlocks, and with bigger sst file size (SetTargetFileSizeMultiplier) and block size (32k), the total amount of filter and index should be reduced.

I remembered the value SetMaxOpenFiles was affecting the RAM usage a lot when exporting the genesis file. >64GB using when the value is 4096 and <32GB using when it is 1000 Maybe we need to evaluate it against the new settings. Before the genesis export feature is settled in the SDK, it will always be an issue.

can you try SetCacheIndexAndFilterBlocks(true) to control ram usage during genesis export? I think the reason open file limit affect ram usage is because the index/filter caches, CacheIndexAndFilterBlocks will manage those caches together with block cache.

tried cronosd1.0.2-3-gca5e564 on the dev machine has 64GB RAM Recent datasets cronosmainnet_25-1-rocksdb-pruned.20230122.2040.tar.lz4, OOM. Will try the same code with limited SetMaxOpenFiles

@yihuang
Copy link
Collaborator Author

yihuang commented Jan 25, 2023

file descriptors should not be a problem, if ram usage is a concern we should enable SetCacheIndexAndFilterBlocks, and with bigger sst file size (SetTargetFileSizeMultiplier) and block size (32k), the total amount of filter and index should be reduced.

I remembered the value SetMaxOpenFiles was affecting the RAM usage a lot when exporting the genesis file. >64GB using when the value is 4096 and <32GB using when it is 1000 Maybe we need to evaluate it against the new settings. Before the genesis export feature is settled in the SDK, it will always be an issue.

can you try SetCacheIndexAndFilterBlocks(true) to control ram usage during genesis export? I think the reason open file limit affect ram usage is because the index/filter caches, CacheIndexAndFilterBlocks will manage those caches together with block cache.

tried cronosd1.0.2-3-gca5e564 on the dev machine has 64GB RAM Recent datasets cronosmainnet_25-1-rocksdb-pruned.20230122.2040.tar.lz4, OOM. Will try the same code with limited SetMaxOpenFiles

Can you try SetCacheIndexAndFilterBlocks(true)?

Ah ,right. Was thinking your changes already enabled this option. will do it.

@yihuang
Copy link
Collaborator Author

yihuang commented Jan 26, 2023

Is genesis export OOM or node syncing OOM? @borischeuk-crypto has tried on one archive node, didn't see much change in runtime statistics.

Tried SetCacheIndexAndFilterBlocks(true), the genesis export OOM.

yihuang added a commit that referenced this pull request Jan 26, 2023
* Problem: eth_sendTransaction is not tested

* Problem: json-rpc apis fail for legacy blocks after upgrade (#696)

* Problem: json-rpc apis fail for legacy blocks

Solution:
- keep the query handler in cosmos-sdk backward-compatible
- add integration test to check

* update sdk to upstream

* ibc-go to rc2

* Problem: file changes detection in workflow is problematic (backport #703) (#705)

* Problem: file changes detection in workflow is problematic

Solution:
- fix wildcards according the plugin's doc
- reformat python

* fix py-lint

* Problem: after v0.9.0 upgrade eth_call failed on old blocks (backport #713) (#719)

* Problem: after v0.9.0 upgrade eth_call failed on old blocks

Solution:
- make grpc query compatible with old format

* debug

* fix eth_call

* fix gravity upgrade test

* update ethermint to main branch

* update sdk

* Problem: state streamers are not integrated (backport #702) (#721)

Solution:
- integration the basic file streamer

* add integration test

* changelog

* fix build

* fix lint

* fix deliver tx event in cosmos-sdk

* fix integration test

* Update integration_tests/test_streamer.py

Signed-off-by: yihuang <huang@crypto.com>

* update ethermint and fix build

* add a small cli utility into test_streamer.py

* fix integration test

* update sdk to upstream

Signed-off-by: yihuang <huang@crypto.com>

Signed-off-by: yihuang <huang@crypto.com>

* Problem: new iavl indexes migration is slow and not optional (#714) (#720)

* Problem: new iavl indexes migration is slow and not optional

Closes: #712
Solution:
- Integrate the option introduced in cosmos-sdk

* Update CHANGELOG.md

Signed-off-by: yihuang <huang@crypto.com>

Signed-off-by: yihuang <huang@crypto.com>

Signed-off-by: yihuang <huang@crypto.com>

* Problem: recent dependencies are not used (backport #729) (#730)

* Problem: recent dependencies are not used (backport #729)

Solution:
- update cosmos-sdk to 0.46.2, ibc-go to v5.0.0, ethermint to recent main branch

Update highlights:
- new flag to disable fast node migration
- fix streaming listeners bug
- fix grpc server panic
- fix index-eth-tx error on empty db

* Update CHANGELOG.md

Signed-off-by: yihuang <huang@crypto.com>

Signed-off-by: yihuang <huang@crypto.com>

* Problem: chain state is inconsistent if upgrade migration is interrupted (#748)

* Problem: chain state is inconsistent if upgrade migration is interrupted

Solution:
- update cosmos-sdk with the fix

* Update CHANGELOG.md

Signed-off-by: yihuang <huang@crypto.com>

* gomod2nix

* skip streamer test

Signed-off-by: yihuang <huang@crypto.com>

* Problem: recent fixes in dependencies are not included (#752)

* Problem: recent fixes in dependencies are not included

Solution:
- update cosmos-sdk and iavl

* Update CHANGELOG.md

Signed-off-by: yihuang <huang@crypto.com>

* fix build

Signed-off-by: yihuang <huang@crypto.com>

* Problem: binary version is not bump to v1.0.0 (#753)

* Problem: recent fixes in dependencies are not used (#757)

* Problem: recent fixes in dependencies are not used

Solution:
- cosmos-sdk -> v0.46.4
- ethermint -> main
- ibc-go -> v5.0.1
- add dragonberry ics20 replacement

* maintain ethermint fork

* Problem: gas used is not backward compatible (#760)

Solution:
- revert the changes in ethermint

* Problem: evm execute result is non-deterministic with concurrent grpc query (#761)

* Problem: evm execute result is non-deterministic with concurrent grpc query

Solution:
- update dependencies to include the fix

* Update CHANGELOG.md

Signed-off-by: yihuang <huang@crypto.com>

* Update go.mod

Signed-off-by: yihuang <huang@crypto.com>

Signed-off-by: yihuang <huang@crypto.com>

* Problem: extra_eips is not cleared on production network (#762)

* Problem: extra_eips is not cleared on production network

Closes: #755
Solution:
- add 1.0.0 upgrade plan to clear it

* Update CHANGELOG.md

Signed-off-by: yihuang <huang@crypto.com>

* Update integration_tests/test_upgrade.py

Signed-off-by: yihuang <huang@crypto.com>

* fix integration test

Signed-off-by: yihuang <huang@crypto.com>

* Problem: no error log when iavl set failure trigger app hash mismatch (#763)

* Problem: no error log when iavl set failure trigger app hash mismatch

Solution:
- log the error in cosmos-sdk

* Update CHANGELOG.md

Signed-off-by: yihuang <huang@crypto.com>

* PR merged

Signed-off-by: yihuang <huang@crypto.com>

* Problem: different result from eth_getProof comparing with Ethereum (#764)

* Problem: different result from eth_getProof comparing with Ethereum

Solution:
- cherry-pick solution from ethermint, thanks @mmsqe

* Update CHANGELOG.md

Signed-off-by: yihuang <huang@crypto.com>

Signed-off-by: yihuang <huang@crypto.com>

* Problem: nix exceeds github rate limit occationally in CI (backport #766) (#768)

Solution:
- configure access-token
- update the action plugins

* Problem: fixes in ibc-go v5.1 are not included (#765)

* Problem: fixes in ibc-go v5.1 are not included

Solution:
- make a breaking change to upgrade to ibc-go `v5.1.0`.
- will do v1.0.0 upgrade on both testnet and mainnet.

* Update CHANGELOG.md

Signed-off-by: yihuang <huang@crypto.com>

* fix lint

* include cache fix in tendermint

* update sdk

* Update CHANGELOG.md

Signed-off-by: yihuang <huang@crypto.com>

* Update CHANGELOG.md

Signed-off-by: yihuang <huang@crypto.com>

* make different plan name v1.0.0-testnet3 for testnet3

Signed-off-by: yihuang <huang@crypto.com>
Co-authored-by: mmsqe <mavis@crypto.com>

* Problem: london hardfork number failed validation (#771)

* fix upgrade set parameters

* changelog

* Problem: formal v0.46.5 cosmos-sdk release is not used (#772)

* Problem: formal v0.46.5 cosmos-sdk release is not used

Solution:
- update dependency, should be non-breaking for cronos

* Update CHANGELOG.md

Signed-off-by: yihuang <huang@crypto.com>

* update to v0.46.6

Signed-off-by: yihuang <huang@crypto.com>

* Problem: final v1.0.0 is not released (#774)

Solution:
- update changelog

* Problem: manual prune cmd is not included (backport #781) (#782)

Solution:
- add to root cmd

* Problem: cosmos-sdk `v0.46.7` is not used (#790)

* Problem: cosmos-sdk `v0.46.7` is not used

Solution:
- update dependency
- `v0.46.7` fix a gov migration issue which affect query votes of old proposals.

* Update CHANGELOG.md

Signed-off-by: yihuang <huang@crypto.com>

* use sdk streamers config

* fix streamer test

* fix file streamer integration test

* changelog

Signed-off-by: yihuang <huang@crypto.com>

* Problem: discontinued ibc-go version (#802)

* Problem: discontinued ibc-go version

Solution:
- update ibc-go to v5.2.0.
- do another coordinated upgrade on testnet3.

* Update CHANGELOG.md

Signed-off-by: yihuang <huang@crypto.com>

* Update app/upgrades.go

Signed-off-by: yihuang <huang@crypto.com>

Signed-off-by: yihuang <huang@crypto.com>

* Problem: production rocksdb configuration is not optimal (#813)

* Problem: production rocksdb configuration is not optimal

Solution:
- update related dependencies to allow customize rocksdb options.
- especially using rocksdb v7.
- tune rocksdb options.

* Update Makefile

Signed-off-by: yihuang <huang@crypto.com>

* remove rocksdb from niv

* rocksdb options

* update flake

* fix build

* create_if_missing

* OptimizeLevelStyleCompaction and IncreaseParallelism

* remove SetLevelCompactionDynamicLevelBytes and add BlockCache

* fix integration test

* comments

Signed-off-by: yihuang <huang@crypto.com>

* Problem: prometheus metrics is lost (#814)

* Problem: prometheus metrics is lost

Solution:
- setup correctly in ethermint

* changelog

* release v1.0.3

* Update CHANGELOG.md

Signed-off-by: yihuang <huang@crypto.com>

* fix changelog

* fix merge

* Update integration_tests/test_upgrade.py

Co-authored-by: mmsqe <tqd0800210105@gmail.com>
Signed-off-by: yihuang <huang@crypto.com>

* fix test

* Update integration_tests/configs/default.jsonnet

Signed-off-by: yihuang <huang@crypto.com>

* fix test_multiple_attestation_processing

* fix changelog

Signed-off-by: yihuang <huang@crypto.com>
Co-authored-by: mmsqe <mavis@crypto.com>
Co-authored-by: mmsqe <tqd0800210105@gmail.com>
Co-authored-by: Tomas Tauber <2410580+tomtau@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants