Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

check export snapshot status with the manifest file #3561

Merged
merged 4 commits into from
Jul 25, 2022

Conversation

yourmoonlight
Copy link
Collaborator

Pull request type

Please check the type of change your PR introduces:

  • Bugfix
  • Feature
  • Code style update (formatting, renaming)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • Documentation content changes
  • Other (please describe):

What is the current behavior?

Currently the export snapshot process may be failed for some unknown reason, so we should add some check logic to make the snapshot is correct for sure.

Other information

#3539

@github-actions
Copy link

Benchmark for 7db8e4e

Click to view benchmark
Test Base PR %
accumulator_append 602.9±11.18µs 606.8±9.95µs +0.65%
block_apply/block_apply_10 264.8±0.60ms 268.2±0.83ms +1.28%
block_apply/block_apply_1000 27.0±0.02s 27.1±0.02s +0.37%
get_with_proof/db_store 36.3±0.21µs 36.4±0.40µs +0.28%
get_with_proof/mem_store 31.4±0.77µs 31.3±0.15µs -0.32%
put_and_commit/db_store/1 95.4±4.94µs 94.9±5.09µs -0.52%
put_and_commit/db_store/10 850.5±39.36µs 866.4±54.86µs +1.87%
put_and_commit/db_store/100 7.4±0.32ms 7.3±0.32ms -1.35%
put_and_commit/db_store/5 444.3±23.75µs 437.3±22.99µs -1.58%
put_and_commit/db_store/50 3.8±0.17ms 3.8±0.17ms 0.00%
put_and_commit/mem_store/1 61.7±5.59µs 61.7±5.65µs 0.00%
put_and_commit/mem_store/10 575.5±47.22µs 579.5±47.19µs +0.70%
put_and_commit/mem_store/100 5.7±0.83ms 5.7±0.82ms 0.00%
put_and_commit/mem_store/5 291.1±24.79µs 292.9±24.33µs +0.62%
put_and_commit/mem_store/50 2.9±0.18ms 2.8±0.19ms -3.45%
query_block/query_block_in(10)_times(100) 3.8±0.13ms 3.9±0.07ms +2.63%
query_block/query_block_in(10)_times(1000) 38.4±0.84ms 39.4±0.97ms +2.60%
query_block/query_block_in(10)_times(10000) 388.3±8.16ms 387.8±8.86ms -0.13%
query_block/query_block_in(1000)_times(100) 941.7±5.62µs 937.2±7.08µs -0.48%
query_block/query_block_in(1000)_times(1000) 9.4±0.10ms 9.4±0.10ms 0.00%
query_block/query_block_in(1000)_times(10000) 93.6±0.80ms 93.6±0.51ms 0.00%
storage_transaction 1043.8±268.90µs 1064.8±285.02µs +2.01%
vm/transaction_execution/1 315.0±1.79ms 311.7±0.63ms -1.05%
vm/transaction_execution/10 99.5±1.73ms 98.3±0.22ms -1.21%
vm/transaction_execution/20 92.6±0.77ms 91.7±0.38ms -0.97%
vm/transaction_execution/5 120.8±0.36ms 120.6±1.13ms -0.17%
vm/transaction_execution/50 106.7±0.55ms 106.1±0.36ms -0.56%

os.system(export_snapshot_cmd)
export_status = os.system(
"bash -c \"if [ $(less /sc-data/snapshot/manifest.csv| grep state_node | awk -F ' ' '{print$2}') -eq $(less /sc-data/snapshot/state_node | wc -l) ]; then exit 0; else exit 1;fi\"")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里五个文件都的检测下

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里五个文件都的检测下

好的。

@nkysg nkysg self-requested a review July 25, 2022 14:14
@codecov
Copy link

codecov bot commented Jul 25, 2022

Codecov Report

Merging #3561 (74720f6) into master (9f5d1c1) will increase coverage by 0.24%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3561      +/-   ##
==========================================
+ Coverage   29.12%   29.35%   +0.24%     
==========================================
  Files         589      589              
  Lines       49753    49805      +52     
  Branches    23411    23433      +22     
==========================================
+ Hits        14486    14617     +131     
- Misses      20842    20850       +8     
+ Partials    14425    14338      -87     
Flag Coverage Δ
unittests 29.35% <ø> (+0.24%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
txpool/src/pool/listener/tests.rs 10.91% <0.00%> (-3.63%) ⬇️
network-rpc/api/src/lib.rs 20.66% <0.00%> (-3.26%) ⬇️
consensus/src/difficulty.rs 40.00% <0.00%> (-2.35%) ⬇️
network-rpc/src/rpc.rs 19.57% <0.00%> (-2.17%) ⬇️
config/src/rpc_config.rs 42.60% <0.00%> (-2.16%) ⬇️
storage/src/db_storage/mod.rs 40.00% <0.00%> (-2.12%) ⬇️
config/src/helper.rs 41.18% <0.00%> (-1.96%) ⬇️
config/src/metrics_config.rs 50.00% <0.00%> (-1.96%) ⬇️
vm/types/src/token/token_value.rs 55.56% <0.00%> (-1.38%) ⬇️
...etwork-p2p/src/protocol/generic_proto/behaviour.rs 21.79% <0.00%> (-1.21%) ⬇️
... and 81 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7bbf1d4...74720f6. Read the comment docs.

@github-actions
Copy link

Benchmark for 87c572c

Click to view benchmark
Test Base PR %
accumulator_append 604.3±11.93µs 606.4±11.92µs +0.35%
block_apply/block_apply_10 264.3±0.61ms 265.3±0.47ms +0.38%
block_apply/block_apply_1000 27.1±0.02s 27.2±0.03s +0.37%
get_with_proof/db_store 36.1±0.24µs 36.5±0.28µs +1.11%
get_with_proof/mem_store 31.4±0.17µs 31.5±0.43µs +0.32%
put_and_commit/db_store/1 94.9±4.83µs 95.7±6.86µs +0.84%
put_and_commit/db_store/10 854.6±38.79µs 858.0±38.83µs +0.40%
put_and_commit/db_store/100 7.3±0.33ms 7.4±0.32ms +1.37%
put_and_commit/db_store/5 435.3±22.02µs 433.5±21.95µs -0.41%
put_and_commit/db_store/50 3.8±0.17ms 3.8±0.17ms 0.00%
put_and_commit/mem_store/1 61.9±5.60µs 61.9±5.63µs 0.00%
put_and_commit/mem_store/10 578.8±47.04µs 581.1±49.24µs +0.40%
put_and_commit/mem_store/100 5.6±0.34ms 5.7±0.83ms +1.79%
put_and_commit/mem_store/5 292.0±24.29µs 293.6±25.73µs +0.55%
put_and_commit/mem_store/50 2.8±0.18ms 2.8±0.18ms 0.00%
query_block/query_block_in(10)_times(100) 3.9±0.13ms 3.9±0.14ms 0.00%
query_block/query_block_in(10)_times(1000) 39.3±1.46ms 38.7±1.07ms -1.53%
query_block/query_block_in(10)_times(10000) 387.0±7.34ms 390.5±5.05ms +0.90%
query_block/query_block_in(1000)_times(100) 954.6±9.59µs 953.1±6.36µs -0.16%
query_block/query_block_in(1000)_times(1000) 9.6±0.03ms 9.6±0.09ms 0.00%
query_block/query_block_in(1000)_times(10000) 95.6±1.14ms 96.0±1.26ms +0.42%
storage_transaction 1059.6±325.49µs 1045.7±267.80µs -1.31%
vm/transaction_execution/1 312.8±1.40ms 314.3±4.12ms +0.48%
vm/transaction_execution/10 98.5±0.24ms 98.7±0.25ms +0.20%
vm/transaction_execution/20 92.0±0.71ms 92.2±0.63ms +0.22%
vm/transaction_execution/5 120.3±0.72ms 121.2±1.64ms +0.75%
vm/transaction_execution/50 106.3±0.55ms 106.3±0.21ms 0.00%

@jolestar jolestar merged commit 6011f5f into master Jul 25, 2022
@jolestar jolestar deleted the sync_snapshot_with_check branch July 25, 2022 16:01
@nkysg nkysg linked an issue Jul 25, 2022 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug Report] panicked when importing snapshot
3 participants