Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[base-node] Fix handling of large chain reorgs in header sync #2759

Merged
merged 1 commit into from
Mar 31, 2021
Merged

[base-node] Fix handling of large chain reorgs in header sync #2759

merged 1 commit into from
Mar 31, 2021

Conversation

sdbondi
Copy link
Member

@sdbondi sdbondi commented Mar 16, 2021 •
edited
Loading

Description

Changes header sync to continue downloading headers until a higher
accumulated PoW than the current tip is achieved.

Determine chain split hash

image

Download more headers until higher PoW than the current tip
image

Reorg chain (if necessary) once higher accumulated PoW found

image

Continue downloading and validating headers until the complete, adding 1000 at a time to the end of the chain

image

Header sync complete

image

Motivation and Context

Previously, in > 1000 height reorgs, headers at split + 1000 are
compared which is insufficient to determine if the remote chain is
stronger than the local one.

How Has This Been Tested?

New cucumber test Full block sync with (large|small) reorg
New basic unit tests for header validator

Following test cases on stibbons base node:

  • sync from scratch
  • simple sync, node behind ~200 blocks
  • split from genesis (mined from 0, then after 27 blocks, connected to main stibbons network)
  • node that had previously had the erroneous weaker target difficult issue (resulting in bans)
    • large reorg with split ~3500 blocks deep, ~9500 blocks behind
  • resume interrupted sync

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Feature refactor (No new feature or functional changes, but performance or technical debt improvements)
  • New Tests
  • Documentation

Checklist:

  • I'm merging against the development branch.
  • I ran cargo-fmt --all before pushing.
  • I ran cargo test successfully before submitting my PR.
  • I have squashed my commits into a single commit.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.

@sdbondi sdbondi force-pushed the core-allow-sync-reorg-1000 branch 3 times, most recently from 7ed4bd6 to cb9a05f Compare March 17, 2021 12:11
@sdbondi sdbondi changed the title wip [core] Fix handling of large chain reorgs in header sync [core] Fix handling of large chain reorgs in header sync Mar 17, 2021
@sdbondi sdbondi changed the title [core] Fix handling of large chain reorgs in header sync [base-node] Fix handling of large chain reorgs in header sync Mar 17, 2021
@sdbondi sdbondi force-pushed the core-allow-sync-reorg-1000 branch from 49ad3b9 to 40d019e Compare March 17, 2021 12:43
@sdbondi sdbondi force-pushed the core-allow-sync-reorg-1000 branch 2 times, most recently from 1be7823 to 9cd9e3b Compare March 18, 2021 10:40
delta1
delta1 suggested changes Mar 19, 2021
View reviewed changes
Copy link
Contributor

@delta1 delta1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, I'm still trying to get the large reorg integration test done successfully locally - I had to increase the timeout and make that JS change to curTipHash

base_layer/core/src/base_node/sync/header_sync/synchronizer.rs Outdated Show resolved Hide resolved
base_layer/core/src/base_node/sync/header_sync/synchronizer.rs Show resolved Hide resolved
integration_tests/features/support/steps.js Outdated Show resolved Hide resolved
base_layer/core/src/base_node/sync/header_sync/validator.rs Show resolved Hide resolved
@sdbondi sdbondi force-pushed the core-allow-sync-reorg-1000 branch 5 times, most recently from 254c6e7 to baa411f Compare March 22, 2021 09:29
@sdbondi
Copy link
Member Author

sdbondi commented Mar 22, 2021

@delta1 Fixed the typo and disabled timeouts for the mining tasks in favour of small timeouts for each block mined, making it proportional to the number of blocks requested

@sdbondi sdbondi force-pushed the core-allow-sync-reorg-1000 branch 3 times, most recently from b564892 to 7465208 Compare March 23, 2021 05:18
@delta1
Copy link
Contributor

delta1 commented Mar 23, 2021

Nice the integration test passed first time for me - looking good

delta1
delta1 previously approved these changes Mar 23, 2021
View reviewed changes
Copy link
Contributor

@delta1 delta1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - tested

@sdbondi sdbondi force-pushed the core-allow-sync-reorg-1000 branch 5 times, most recently from c05a348 to bc54eee Compare March 30, 2021 12:31
@sdbondi
[core] Fix handling of large chain reorgs in header sync
19cf3b2 
Changes header sync to continue downloading headers until a higher
accumulated PoW than the current tip is acheived.

Added cucumber tests for sync with large reorg
@sdbondi sdbondi force-pushed the core-allow-sync-reorg-1000 branch from bc54eee to 19cf3b2 Compare March 31, 2021 12:48
@stringhandler stringhandler merged commit 0542b93 into tari-project:development Mar 31, 2021
@sdbondi sdbondi deleted the core-allow-sync-reorg-1000 branch March 31, 2021 12:52
stringhandler added a commit that referenced this pull request Apr 1, 2021
@stringhandler
v0.8.7
8076922 
Changes since v0.8.6

Wallet
---
- [#2825](#2825) [wallet] correctly log RPC error messages
- [#2824](#2824) [wallet] Output error message for ConfigError
- [#2818](#2818) [wallet] get_transaction_info no longer errors if any transaction is not found
- [#2819](#2819) [wallet] Clear UI before starting UI loop
- [#2798](#2798) [wallet] Add MinedUnconfirmed to command mode wait stage
- [#2793](#2793) [wallet] Add menu to console wallet
- [#2765](#2765) [wallet] Friendly incorrect password message
- [#2753](#2753) [wallet] Update base node service to use RPC
- [#2762](#2762) [wallet] Implement wallet recovery process in LibWallet FFI
- [#2761](#2761) [wallet] Differentiate "funds pending" case in UTXO selection
- [#2748](#2748) [wallet] Added saf message duration to configuration

Base Node
---
- [#2814](#2814) [base-node] Add previous hash check to sync validator
- [#2759](#2759) [base-node] Fix handling of large chain reorgs in header sync
- [#2813](#2813) [base-node] Add rewind-blockchain and improve header-stats command
- [#2795](#2795) [base-node] Update mempool consensus validator to check weight excluding coinbase
- [#2738](#2738) [base-node] Rewind prune mode behind horizon
- [#2768](#2768) [base-node] Add transaction consensus validator
- [#2764](#2764) [base-node] Remove reorgs from total tx count
- [#2755](#2755) [base-node] Node bootstraps after initial sync if all other nodes are h=0
- [#2751](#2751) [base-node] Minor fix for list-headers

Mining
---
- [#2807](#2807) [merge-mining] Added getLastBlockHeader cucumber test
- [#2792](#2792) [mining-node] Added additional command line arguments
- [#2791](#2791) [merge-mining] Add stdout information
- [#2767](#2767) [mmproxy] Expanded cucumber tests for merged mining proxy
- [#2743](#2743) [merge-mining] Fix logic flaw in proxy_request_to_monerod

Other
---
- [#2826](#2826) [common] Add rpc_max_simultaneous_sessions setting to config
- [#2823](#2823) [chore] Fix import
- [#2822](#2822) [chore] Update prettier paths
- [#2815](#2815) [comms] Connection manager requester not exposed on CommsNode
- [#2820](#2820) [chore] Use prettier for integration tests
- [#2821](#2821) [other] Add memory net summary
- [#2757](#2757) [ci] Add all-targets to clippy
- [#2796](#2796) [docs] Add WalletNotify info to config samples
- [#2773](#2773) [tests] Add custom log file paths for wallet and mmproxy
- [#2763](#2763) [tests] Update mempool waits to be dynamic
- [#2769](#2769) [ci] Increase Cucumber CI to include all tests except "long running"
- [#2747](#2747) [docs] Update rfc 201 for coinbase
- [#276](#276) [chore] Removed some unused dependencies
- [#2576](#2576) [common] Increase flood ban config
- [#2572](#2572) [tests] Add cucumber test verify meddling with MMR sizes
- [#2754](#2754) [common] Emit rerun-if-changed directives for include protos
- [#2745](#2745) [ci] Point Circle CI status badge to development branch only
- [#2744](#2744) [docs] Minor Fixes
- [#2740](#2740) [chore] Remove the text_message_service from the wallet (#2740)
- [#2739](#2739) [docs] Document the NO_OP TariScript vulnerability
@stringhandler stringhandler mentioned this pull request Apr 1, 2021
Merged
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SWvheerden SWvheerden SWvheerden left review comments

@delta1 delta1 delta1 left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sdbondi @delta1 @SWvheerden @stringhandler