Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error: stage encountered an error in block #19247413: validation error: mismatched block state root #6644

Closed
1 task done
krakovia-evm opened this issue Feb 17, 2024 · 13 comments
Closed
1 task done

Error: stage encountered an error in block #19247413: validation error: mismatched block state root #6644

krakovia-evm opened this issue Feb 17, 2024 · 13 comments
Labels
A-staged-sync Related to staged sync (pipelines and stages) A-trie Related to Merkle Patricia Trie implementation C-bug An unexpected or incorrect behavior

Comments

@krakovia-evm
Copy link

Describe the bug

my full node cannot get in sync due to validation error, attached logs
version is .18
The database has been created with ~rev7, > 13 > 18.
Been in sync with the rev18 since it's release

Steps to reproduce

ns

Node logs

2024-02-17T20:53:44.899857Z  INFO Stage finished executing pipeline_stages=5/13 stage=Execution checkpoint=19247472 target=19247472 stage_progress=100.00%
2024-02-17T20:53:44.899922Z  INFO Executing stage pipeline_stages=6/13 stage=MerkleUnwind checkpoint=19247472 target=19247472
2024-02-17T20:53:44.899928Z  INFO Stage is always skipped
2024-02-17T20:53:44.899944Z  INFO Stage finished executing pipeline_stages=6/13 stage=MerkleUnwind checkpoint=19247472 target=19247472 stage_progress=None
2024-02-17T20:53:44.899976Z  INFO Executing stage pipeline_stages=7/13 stage=AccountHashing checkpoint=19247413 target=19247472
2024-02-17T20:53:44.997271Z  INFO Stage finished executing pipeline_stages=7/13 stage=AccountHashing checkpoint=19247472 target=19247472 stage_progress=100.00%
2024-02-17T20:53:45.138359Z  INFO Executing stage pipeline_stages=8/13 stage=StorageHashing checkpoint=19247413 target=19247472
2024-02-17T20:53:45.335738Z  INFO Stage finished executing pipeline_stages=8/13 stage=StorageHashing checkpoint=19247472 target=19247472 stage_progress=100.00%
2024-02-17T20:53:45.601545Z  INFO Executing stage pipeline_stages=9/13 stage=MerkleExecute checkpoint=19247413 target=19247472
2024-02-17T20:53:47.898861Z  INFO Status connected_peers=0 freelist=372593 stage=MerkleExecute checkpoint=19247413 target=19247472 stage_progress=None
2024-02-17T20:53:48.428705Z  WARN Failed to verify block state root target_block=19247472 got=0x8de129ebf8add24309c92f90035ffa051a3c6afae6e58b3f78d15491299b477e expected=SealedHeader { header: Header { parent_hash: 0x32ecfca295830a18c0634abe8c384317d9cec1fb0872a0a084af6c759a752f52, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x95222290dd7278aa3ddd389cc1e1d165cc4bafe5, state_root: 0x802a95c6a53eb91ccf8e42a3347b6da40db5aa34bae178be57eb42179ba15b58, transactions_root: 0x0d07474e6f41e2c88ae12053eb404af86220468df47613ee893fe2980e3d04ca, receipts_root: 0x6dde8a00f420796a474a52ae801eaf270b5967483a1bef6848c0357764841319, withdrawals_root: Some(0x03dc3016117408e5cad37737adeff6e154a522f12e814477aaa3d09ad3395c5e), logs_bloom: 0x01a00e1344120b20106052208212c1615821508021a0714c0b3d01199101340141099a80206084224004480a0011a13c824000a49a462602e209889120bf2c800c14881919319a2e5d00c8698969c2a2638a029501604f0802632001802089c0963ac110a22300038448526308802d420b34004308845401d36001580c090100062013890c0dd528014100403a02901d8601c781abc8a848003d0a78a0540ca2ab002060588838003432c283989235001880a8404c98058260a1802c020148200514c022487a500020184912088c7490134007c0226684d71ce0102a4240304c46532c44b49480004ac468801040100282349410b60a6f4309002620651433c9, difficulty: 0x0_U256, number: 19247472, gas_limit: 30000000, gas_used: 9936927, timestamp: 1708171607, mix_hash: 0xbbf5e70d1353a56b40d6be8b2ab19110847ca620ab9045ea4e1edd7afbb678f4, nonce: 0, base_fee_per_gas: Some(18100826180), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x6265617665726275696c642e6f7267 }, hash: 0x69fed8182e7f5703be741653a2b538a55b6faec99c35569a59ca7809d117a9d4 }
2024-02-17T20:53:48.428911Z ERROR Stage encountered a validation error: mismatched block state root: got 0x8de129ebf8add24309c92f90035ffa051a3c6afae6e58b3f78d15491299b477e, expected 0x802a95c6a53eb91ccf8e42a3347b6da40db5aa34bae178be57eb42179ba15b58 stage=MerkleExecute bad_block=19247472
2024-02-17T20:53:48.518148Z  INFO Unwinding{stage=StorageHashing}: Stage unwound stage=StorageHashing unwind_to=19247413 progress=19247413 done=true
2024-02-17T20:53:49.832012Z  INFO Unwinding{stage=AccountHashing}: Stage unwound stage=AccountHashing unwind_to=19247413 progress=19247413 done=true
2024-02-17T20:53:51.505802Z  WARN Unwinding{stage=MerkleUnwind}: Failed to verify block state root target_block=19247413 got=0xe9be281743d55590dcad4001ec0f6fc5e300c6e88385d15f9de46e23e70bc5e5 expected=SealedHeader { header: Header { parent_hash: 0x1c8c0ebf78eb2442ec85d5fb31a1fc40f4814f9a05e06145a73afb1091edfbaa, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0xdafea492d9c6733ae3d56b7ed1adb60692c98bc5, state_root: 0x9b3978120e83f0a4fac64bf278197a21f1ad9744563ae6c2c9f5ae0533c24bb0, transactions_root: 0x77a8c90d187c66cb93d772dbe1c327b4900817acf439c354b88701bbc201e82d, receipts_root: 0xe0a34a30c10bac70203d12ddb2edeb543b95bcee5073d98b14dc8cc999a180dc, withdrawals_root: Some(0x8da92ff3ad3a0ea33ff7c119c4c16c9d404423061f5bde15c39e12f9c7e38338), logs_bloom: 0x18a3f317a5b96942b012d9438f01d7f3d7498339be35e28af14b1e93bc64d530c49685f390422230e719dc37957de5ce2ea9b444ad00fb1537b5534c41bf871f483e2b48070428eb69f65bafc46c71f57687c21b1c5ade4d091f8f59adfc3c4b7ffd469c3e0e32192dcdde645f706949ba3bc3fbf41ef428f735fd95241e42c806ba18e83ae4908da45d7d7b3bc6420812719601f3b5a86f22e5c9d42fdcfee177433bf10daf6baf7f8170f4fd8ce476b451e711b63e23091bf9a0ae920b07c0583c86022d5bc28bade8de15093fdefe1838675ec640e1d70c1d42ba84feb20974bc7c4dab94a42a890d138ff0597018897ef2195b2866430081e60030011418, difficulty: 0x0_U256, number: 19247413, gas_limit: 30000000, gas_used: 16969930, timestamp: 1708170887, mix_hash: 0x3db1505513b125e93d2d6431cec0d985609a56247b0aba95ff19b2cb4405dba5, nonce: 0, base_fee_per_gas: Some(18664134201), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x496c6c756d696e61746520446d6f63726174697a6520447374726962757465 }, hash: 0xca30438c9d8dcc0b6a70833adaea237722ce79d7eeb5759377910dc6cce31aa5 }
2024-02-17T20:53:51.520445Z ERROR shutting down due to error
Error: stage encountered an error in block #19247413: validation error: mismatched block state root: got 0xe9be281743d55590dcad4001ec0f6fc5e300c6e88385d15f9de46e23e70bc5e5, expected 0x9b3978120e83f0a4fac64bf278197a21f1ad9744563ae6c2c9f5ae0533c24bb0
Caused by:
   0: validation error: mismatched block state root: got 0xe9be281743d55590dcad4001ec0f6fc5e300c6e88385d15f9de46e23e70bc5e5, expected 0x9b3978120e83f0a4fac64bf278197a21f1ad9744563ae6c2c9f5ae0533c24bb0
   1: mismatched block state root: got 0xe9be281743d55590dcad4001ec0f6fc5e300c6e88385d15f9de46e23e70bc5e5, expected 0x9b3978120e83f0a4fac64bf278197a21f1ad9744563ae6c2c9f5ae0533c24bb0
�
:
    /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/task/poll.rs:255:39
2024-02-17T20:54:52.652963Z  INFO reth 0.1.0-alpha.18 (7c93d8efa) starting
2024-02-17T20:54:52.652988Z  INFO Opening database path="/root/.local/share/reth/mainnet/db"
2024-02-17T20:54:52.657331Z  INFO Configuration loaded path="/root/.local/share/reth/mainnet/reth.toml"
2024-02-17T20:54:52.657921Z  INFO Database opened
2024-02-17T20:54:52.658053Z  INFO Starting metrics endpoint addr=0.0.0.0:9001
2024-02-17T20:54:52.658189Z  INFO Pre-merge hard forks (block based):
- Frontier                         @0
- Homestead                        @1150000
- Dao                              @1920000
- Tangerine                        @2463000
- SpuriousDragon                   @2675000
- Byzantium                        @4370000
- Constantinople                   @7280000
- Petersburg                       @7280000
- Istanbul                         @9069000
- MuirGlacier                      @9200000
- Berlin                           @12244000
- London                           @12965000
- ArrowGlacier                     @13773000
- GrayGlacier                      @15050000
Merge hard forks:
- Paris                            @58750000000000000000000 (network is not known to be merged)
Post-merge hard forks (timestamp based):
- Shanghai                         @1681338455
- Cancun                           @1710338135
2024-02-17T20:54:52.704319Z  INFO Transaction pool initialized
2024-02-17T20:54:52.704336Z  INFO Connecting to P2P network
2024-02-17T20:54:52.704408Z  INFO Loading saved peers file=/root/.local/share/reth/mainnet/known-peers.json
2024-02-17T20:54:52.705211Z  INFO Connected to P2P network peer_id=0x819cef1777642008afebbc7606fc0d7ae6464e22ed11c10904d6cb527ae22c8083d92b1f53a7912f41e6e78cd77de8f9f91d79d210a2c453e85ac5de10d9aed1 local_addr=0.0.0.0:30303 enode=enode://819cef1777642008afebbc7606fc0d7ae6464e22ed11c10904d6cb527ae22c8083d92b1f53a7912f41e6e78cd77de8f9f91d79d210a2c453e85ac5de10d9aed1@127.0.0.1:30303
2024-02-17T20:54:52.705455Z  INFO Pruner initialized prune_config=PruneConfig { block_interval: 5, segments: PruneModes { sender_recovery: Some(Full), transaction_lookup: None, receipts: Some(Before(11052984)), account_history: Some(Distance(10064)), storage_history: Some(Distance(10064)), receipts_log_filter: ReceiptsLogPruneConfig({0x00000000219ab540356cbb839cbe05303d7705fa: Before(11052984)}) } }
2024-02-17T20:54:52.705660Z  INFO Consensus engine initialized
2024-02-17T20:54:52.705681Z  INFO Engine API handler initialized
2024-02-17T20:54:52.707746Z  INFO RPC auth server started url=0.0.0.0:8551
2024-02-17T20:54:52.707834Z  INFO RPC IPC server started url=/tmp/reth.ipc
2024-02-17T20:54:52.707855Z  INFO RPC HTTP server started url=0.0.0.0:8545
2024-02-17T20:54:52.707857Z  INFO RPC WS server started url=0.0.0.0:8546
2024-02-17T20:54:52.707865Z  INFO Starting consensus engine
2024-02-17T20:54:52.708123Z  INFO Target block already reached checkpoint=19247472 target=Hash(0x69fed8182e7f5703be741653a2b538a55b6faec99c35569a59ca7809d117a9d4)
2024-02-17T20:54:52.708200Z  INFO Executing stage pipeline_stages=1/13 stage=Headers checkpoint=19247472 target=None
2024-02-17T20:54:52.708316Z  INFO Stage finished executing pipeline_stages=1/13 stage=Headers checkpoint=19247472 target=None stage_progress=100.00%
2024-02-17T20:54:52.708458Z  INFO Executing stage pipeline_stages=2/13 stage=TotalDifficulty checkpoint=19247472 target=19247472
2024-02-17T20:54:52.708485Z  INFO Stage finished executing pipeline_stages=2/13 stage=TotalDifficulty checkpoint=19247472 target=19247472 stage_progress=100.00%
2024-02-17T20:54:52.708511Z  INFO Executing stage pipeline_stages=3/13 stage=Bodies checkpoint=19247472 target=19247472
2024-02-17T20:54:52.708534Z  INFO Stage finished executing pipeline_stages=3/13 stage=Bodies checkpoint=19247472 target=19247472 stage_progress=100.00%
2024-02-17T20:54:52.708624Z  INFO Executing stage pipeline_stages=4/13 stage=SenderRecovery checkpoint=19247472 target=19247472
2024-02-17T20:54:52.708650Z  INFO Stage finished executing pipeline_stages=4/13 stage=SenderRecovery checkpoint=19247472 target=19247472 stage_progress=100.00%
2024-02-17T20:54:52.708675Z  INFO Executing stage pipeline_stages=5/13 stage=Execution checkpoint=19247472 target=19247472
2024-02-17T20:54:52.708686Z  INFO Stage finished executing pipeline_stages=5/13 stage=Execution checkpoint=19247472 target=19247472 stage_progress=100.00%
2024-02-17T20:54:52.708771Z  INFO Stage is always skipped
2024-02-17T20:54:52.708793Z  INFO Executing stage pipeline_stages=6/13 stage=MerkleUnwind checkpoint=19247472 target=19247472
2024-02-17T20:54:52.708798Z  INFO Stage finished executing pipeline_stages=6/13 stage=MerkleUnwind checkpoint=19247472 target=19247472 stage_progress=None
2024-02-17T20:54:52.708833Z  INFO Executing stage pipeline_stages=7/13 stage=AccountHashing checkpoint=19247413 target=19247472
2024-02-17T20:54:52.805326Z  INFO Stage finished executing pipeline_stages=7/13 stage=AccountHashing checkpoint=19247472 target=19247472 stage_progress=100.00%
2024-02-17T20:54:52.986452Z  INFO Executing stage pipeline_stages=8/13 stage=StorageHashing checkpoint=19247413 target=19247472
2024-02-17T20:54:53.192792Z  INFO Stage finished executing pipeline_stages=8/13 stage=StorageHashing checkpoint=19247472 target=19247472 stage_progress=100.00%
2024-02-17T20:54:53.533280Z  INFO Executing stage pipeline_stages=9/13 stage=MerkleExecute checkpoint=19247413 target=19247472
2024-02-17T20:54:55.707226Z  INFO Status connected_peers=0 freelist=372593 stage=MerkleExecute checkpoint=19247413 target=19247472 stage_progress=None
2024-02-17T20:54:57.203009Z  WARN Failed to verify block state root target_block=19247472 got=0x8de129ebf8add24309c92f90035ffa051a3c6afae6e58b3f78d15491299b477e expected=SealedHeader { header: Header { parent_hash: 0x32ecfca295830a18c0634abe8c384317d9cec1fb0872a0a084af6c759a752f52, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x95222290dd7278aa3ddd389cc1e1d165cc4bafe5, state_root: 0x802a95c6a53eb91ccf8e42a3347b6da40db5aa34bae178be57eb42179ba15b58, transactions_root: 0x0d07474e6f41e2c88ae12053eb404af86220468df47613ee893fe2980e3d04ca, receipts_root: 0x6dde8a00f420796a474a52ae801eaf270b5967483a1bef6848c0357764841319, withdrawals_root: Some(0x03dc3016117408e5cad37737adeff6e154a522f12e814477aaa3d09ad3395c5e), logs_bloom: 0x01a00e1344120b20106052208212c1615821508021a0714c0b3d01199101340141099a80206084224004480a0011a13c824000a49a462602e209889120bf2c800c14881919319a2e5d00c8698969c2a2638a029501604f0802632001802089c0963ac110a22300038448526308802d420b34004308845401d36001580c090100062013890c0dd528014100403a02901d8601c781abc8a848003d0a78a0540ca2ab002060588838003432c283989235001880a8404c98058260a1802c020148200514c022487a500020184912088c7490134007c0226684d71ce0102a4240304c46532c44b49480004ac468801040100282349410b60a6f4309002620651433c9, difficulty: 0x0_U256, number: 19247472, gas_limit: 30000000, gas_used: 9936927, timestamp: 1708171607, mix_hash: 0xbbf5e70d1353a56b40d6be8b2ab19110847ca620ab9045ea4e1edd7afbb678f4, nonce: 0, base_fee_per_gas: Some(18100826180), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x6265617665726275696c642e6f7267 }, hash: 0x69fed8182e7f5703be741653a2b538a55b6faec99c35569a59ca7809d117a9d4 }
2024-02-17T20:54:57.203188Z ERROR Stage encountered a validation error: mismatched block state root: got 0x8de129ebf8add24309c92f90035ffa051a3c6afae6e58b3f78d15491299b477e, expected 0x802a95c6a53eb91ccf8e42a3347b6da40db5aa34bae178be57eb42179ba15b58 stage=MerkleExecute bad_block=19247472
2024-02-17T20:54:57.337186Z  INFO Unwinding{stage=StorageHashing}: Stage unwound stage=StorageHashing unwind_to=19247413 progress=19247413 done=true
2024-02-17T20:54:58.562271Z  INFO Unwinding{stage=AccountHashing}: Stage unwound stage=AccountHashing unwind_to=19247413 progress=19247413 done=true
2024-02-17T20:54:59.397536Z  INFO Forkchoice updated head_block_hash=0x6b5b1a49a1e3d2eb46c412b6b5c46507f0e31924d83e987b635a3ca72ea5fcba safe_block_hash=0xd1a5059ce35f50710804df3cf6db8ae43dd36dd8489e09132b23d7d711321ed5 finalized_block_hash=0x5c37c1e252e35e03f46ec049f30adef763f7846435b408c36bc08522be842140 status=Syncing
2024-02-17T20:55:00.388872Z  INFO Forkchoice updated head_block_hash=0x2bda8dc35792127ce1a3a3f76fee85f52bf26063ec2fc4ffc5cb0d74f8ce442a safe_block_hash=0xd1a5059ce35f50710804df3cf6db8ae43dd36dd8489e09132b23d7d711321ed5 finalized_block_hash=0x5c37c1e252e35e03f46ec049f30adef763f7846435b408c36bc08522be842140 status=Syncing
2024-02-17T20:55:00.658283Z  WARN Unwinding{stage=MerkleUnwind}: Failed to verify block state root target_block=19247413 got=0xe9be281743d55590dcad4001ec0f6fc5e300c6e88385d15f9de46e23e70bc5e5 expected=SealedHeader { header: Header { parent_hash: 0x1c8c0ebf78eb2442ec85d5fb31a1fc40f4814f9a05e06145a73afb1091edfbaa, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0xdafea492d9c6733ae3d56b7ed1adb60692c98bc5, state_root: 0x9b3978120e83f0a4fac64bf278197a21f1ad9744563ae6c2c9f5ae0533c24bb0, transactions_root: 0x77a8c90d187c66cb93d772dbe1c327b4900817acf439c354b88701bbc201e82d, receipts_root: 0xe0a34a30c10bac70203d12ddb2edeb543b95bcee5073d98b14dc8cc999a180dc, withdrawals_root: Some(0x8da92ff3ad3a0ea33ff7c119c4c16c9d404423061f5bde15c39e12f9c7e38338), logs_bloom: 0x18a3f317a5b96942b012d9438f01d7f3d7498339be35e28af14b1e93bc64d530c49685f390422230e719dc37957de5ce2ea9b444ad00fb1537b5534c41bf871f483e2b48070428eb69f65bafc46c71f57687c21b1c5ade4d091f8f59adfc3c4b7ffd469c3e0e32192dcdde645f706949ba3bc3fbf41ef428f735fd95241e42c806ba18e83ae4908da45d7d7b3bc6420812719601f3b5a86f22e5c9d42fdcfee177433bf10daf6baf7f8170f4fd8ce476b451e711b63e23091bf9a0ae920b07c0583c86022d5bc28bade8de15093fdefe1838675ec640e1d70c1d42ba84feb20974bc7c4dab94a42a890d138ff0597018897ef2195b2866430081e60030011418, difficulty: 0x0_U256, number: 19247413, gas_limit: 30000000, gas_used: 16969930, timestamp: 1708170887, mix_hash: 0x3db1505513b125e93d2d6431cec0d985609a56247b0aba95ff19b2cb4405dba5, nonce: 0, base_fee_per_gas: Some(18664134201), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x496c6c756d696e61746520446d6f63726174697a6520447374726962757465 }, hash: 0xca30438c9d8dcc0b6a70833adaea237722ce79d7eeb5759377910dc6cce31aa5 }
2024-02-17T20:55:00.721298Z ERROR shutting down due to error
2024-02-17T20:55:00.721664Z  INFO Wrote network peers to file peers_file="/root/.local/share/reth/mainnet/known-peers.json"
Error: stage encountered an error in block #19247413: validation error: mismatched block state root: got 0xe9be281743d55590dcad4001ec0f6fc5e300c6e88385d15f9de46e23e70bc5e5, expected 0x9b3978120e83f0a4fac64bf278197a21f1ad9744563ae6c2c9f5ae0533c24bb0
Caused by:
   0: validation error: mismatched block state root: got 0xe9be281743d55590dcad4001ec0f6fc5e300c6e88385d15f9de46e23e70bc5e5, expected 0x9b3978120e83f0a4fac64bf278197a21f1ad9744563ae6c2c9f5ae0533c24bb0
   1: mismatched block state root: got 0xe9be281743d55590dcad4001ec0f6fc5e300c6e88385d15f9de46e23e70bc5e5, expected 0x9b3978120e83f0a4fac64bf278197a21f1ad9744563ae6c2c9f5ae0533c24bb0
�
:
    /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/task/poll.rs:255:39

Platform(s)

Linux (x86)

What version/commit are you on?

reth 0.1.0-alpha.18 (7c93d8e)

What database version are you on?

1

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

docker build .

Code of Conduct

  • I agree to follow the Code of Conduct
@krakovia-evm krakovia-evm added C-bug An unexpected or incorrect behavior S-needs-triage This issue needs to be labelled labels Feb 17, 2024
@krakovia-evm
Copy link
Author

got synced but lasted only some hours.
Logs are filled with those info

2024-02-19T08:16:51.163668Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0xf8ff7180aa953d31945ca2ba0c0a787f65f6278c7fb5b70ebfea001a93d76454 retries=2
2024-02-19T08:16:51.163962Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0xcdec…d7a6" hash=0x00fee01d7d7e7c695ccce0d1ff0e844adde3d1d2806e9bce4823c6e5c3ec657f size=757 previously_seen_size=760 client_version=Nethermind/v1.21.0+bb9b72c0/linux-x64/dotnet7.0.11
2024-02-19T08:16:51.165311Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0x157462f28c97e8cf673deff5781ac9f5409a91992e6b58ab063ba8366ce65385 retries=2
2024-02-19T08:16:51.165315Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0x0521d36bd30e79d2a7410f081a990d1297217b5abb38cde8563ea6693ebabacd retries=2
2024-02-19T08:16:51.165316Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0x170e8852fc8efafec07b84eef50444c3d5ea6518b0a411b5d113a3c7486fae33 retries=2
2024-02-19T08:16:51.165318Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0x1ff5f30797a71f20c6a3226e6e9f68e43924fbff1dbbc0741b73b3c4f45af0e7 retries=2
2024-02-19T08:16:51.165318Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0xa8e4b9883cc40ec2a03ff3aa9d49776dee8e42f7338a2af0b3bfc831952b8701 retries=2
2024-02-19T08:16:51.165319Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0xbc5aab61cfdb3f5abc63dfe6f6371a21d1185097b76c1cc3b5d7ee505ef96ea1 retries=2
2024-02-19T08:16:51.165320Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0x6dea34c2a1ace8b97d1819134247d6a9f46bd8a650d79aebb79b18da5158816b retries=2
2024-02-19T08:16:51.165320Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0xe4c8f67422629be67ed07269fe8836bf6c435304c8cb4ac6557e53e0c4089664 retries=2
2024-02-19T08:16:51.177709Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0xa0ee…9f96" hash=0x018b6305c5e46a6c42aa1ff32f7367fd6c222c91c879b8d36d28629ef82d08c2 size=146 previously_seen_size=148 client_version=Geth/v1.13.12-stable-02eb36af/linux-arm64/go1.21.6
2024-02-19T08:16:51.179342Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0x235de1a4ffd0e0e4b200ced7debd6b05c522651ed8f52b3df89ca8f703cdd713 retries=2
2024-02-19T08:16:51.179347Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0xb46e120f6f9a95c1e62074f5e1263bb746e5593725f85d6cec25c8d6ee707964 retries=2
2024-02-19T08:16:51.179348Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0x44964cf75003fc9a13385f595d9d69f47549bcba7a3055caefc793a2e1bb9e4a retries=2
2024-02-19T08:16:51.179350Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0xde92bca37990f2c9c79674b0e786e9c7d7f8dfa3da0be52791601215debed0a0 retries=2
2024-02-19T08:16:51.179351Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0xdae9ddbe93ac18189b878c456c2fd88fefc3928ab21a91832221681866751fa9 retries=2
2024-02-19T08:16:51.179352Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0x5241b67a902b3c0f737bfa3cc958534f4d35c84c9e78b067c61696a125923fe7 retries=2
2024-02-19T08:16:51.179353Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0x6b5203191abf8b9f2a832e3b1e725f7e9e795aa1b3647d1a266b114ad9556e77 retries=2
2024-02-19T08:16:51.179354Z DEBUG net::tx: retry limit for `GetPooledTransactions` requests reached for hash, dropping hash hash=0xdc747306d34f82c5b1277b4914e48e58c736fe6e11df932f605b705ef3a6d45a retries=2
2024-02-19T08:16:51.180269Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0xeff9…fdf9" hash=0xb60723f916118750689771123b264f83e02e09f4300d336a3c60f61eec2e38e6 size=123 previously_seen_size=121 client_version=erigon/v2.42.0-stable-beb97784/linux-amd64/go1.18.1
2024-02-19T08:16:51.180445Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0xafb1…138b" hash=0x7656f3304965020de48fb2ba181dedbd0ac6d792efedce5a0dbb3bfd41fdc4e3 size=122 previously_seen_size=124 client_version=Geth/v1.13.12-stable-02eb36af/linux-amd64/go1.21.6
2024-02-19T08:16:51.182119Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0xcf54…a585" hash=0xb60723f916118750689771123b264f83e02e09f4300d336a3c60f61eec2e38e6 size=121 previously_seen_size=123 client_version=reth/v0.1.0-alpha.13-b34b0d3/x86_64-unknown-linux-gnu

@krakovia-evm
Copy link
Author

updated to rev19, same error

2024-02-20T00:25:17.244048Z  INFO reth_node_core::events::node: Executing stage pipeline_stages=5/13 stage=Execution checkpoint=19259458 target=19262463
2024-02-20T00:25:19.987572Z DEBUG disc::dns: No dns entry domain=all.mainnet.ethdisco.net hash="3PRKAD6GK24SRD27UY3NZ7"
2024-02-20T00:25:20.659105Z DEBUG net::session: timed out outgoing request id=79 remote_peer_id=0x173393d68fe096ff90ed5e13c136f23fe12205e7c07182f2cabde2aa55719396072b0c6c3c82c17a1d346e7d0c7fb16fb1ace8e43db6c6982612e720278986fc
2024-02-20T00:25:21.800471Z DEBUG discv4: evicting nodes due to failed pong num=28
2024-02-20T00:25:21.800481Z DEBUG discv4: removed node node_id=0x6d4c2dc204fa8e149d6f9b72db749a4260177264c4596638a3a3996decbede3809ae18db79f6472d89fc3118dc3fb9b56ff250191e1b239ba51ca70c8631f3c1
2024-02-20T00:25:21.800486Z DEBUG discv4: removed node node_id=0x651c044839971ca94637938d72377f980a2c72096020b786205bf35ab98bd02b69807d5209ea408b9051d59cff4635c4fa404bb56a0a2d10d91a75abed14cd6a
2024-02-20T00:25:21.800488Z DEBUG discv4: removed node node_id=0xd4e09e6c89b879b4814f466d5bd175a6a2466baf29e97fe89832f9043ef91569eb7ac6ee4455da89ef78d14618c2e8eb96d788d68bbc1ef33f674be7a0c936f8
2024-02-20T00:25:21.800489Z DEBUG discv4: removed node node_id=0x78621bc46ece1da7a8a23154ec8fece8b59723f107713b19423d49cab23c0f25159fb5973e511f7f94b484f1c80bbf885f2520b48f11545bd4487c0b0c24ea4f
2024-02-20T00:25:21.800490Z DEBUG discv4: removed node node_id=0xc3ddabcce1834b919cc274eb50f649467eb273869e955c4842f483421f760a8c4464d8c3774c0dd24f293529ff8d6a790d8f806620f41dda4aea8e6bbbb6cba1
2024-02-20T00:25:21.800492Z DEBUG discv4: removed node node_id=0x5c282e4085e87e8e1b412f6b7f1bc74df596322b3a63b02c4c99f4f6adecc5e3fc7fe146e922cf72ffa784946ef3e5cffb572002a18811056900e7914add702b
2024-02-20T00:25:21.800493Z DEBUG discv4: removed node node_id=0xad5d27b7bf12c0a134d59f2ea70bef395a8422fe0288eca9a7af6fcc9c55451dc870cab2e343029c0defda15a4e138a02bf6c2c38cd506e70c5c0d3fbca6bd36
2024-02-20T00:25:21.800494Z DEBUG discv4: removed node node_id=0x17bf36e5971c03429284d7f3a6873acd6839026490f4fb2c50f9b6faa34a8c3512956503c735dc1dc10efd426a2116e2563ebbc95f3cb8e407080a321a03d46d
2024-02-20T00:25:21.800513Z DEBUG discv4: removed node node_id=0x3e85cd5d26cdf4c2e0d35a0fa4fd02e10b0aad972f29780b1b00b837640e4e3934ddadf0cf209484a1fd5b2a4b2559c87ebd98b1929449b1854ce8bb54b0e867
2024-02-20T00:25:21.800514Z DEBUG discv4: removed node node_id=0x149d38085b7e3fdf4c5cf1d79904ab576a2addc95eb5d829e56f3d3a6b9b9d846e8b4807b6167e09e3c15fb9e15c747de4d930cb1e03848c9031d07043d32b30
2024-02-20T00:25:21.800515Z DEBUG discv4: removed node node_id=0x1571cf3ac74aecba98f010205edaab54a2513085b361c41933b24a6bd6fa56d41eca29c451b6b3db602e227c69d27aae396407a1d89795b5e17238a81760bbb1
2024-02-20T00:25:21.800517Z DEBUG discv4: removed node node_id=0x6febf89f9d4d9f0e5eab3406acea41e975f8ef52f9b55bd344037ca1d8384438da4aa0673dcc23f29fa234e3ac1a9b8f8d6486d4a26c7229dc837c5f131d3bc2
2024-02-20T00:25:21.800518Z DEBUG discv4: removed node node_id=0xb6c14f06b1d287c827b6d255692863b7f12498c1498a094338cea3f441c70762daac9fb2c819f321e4217ed253f348be4c0f24178d2d46629d7bf0093d862d97
2024-02-20T00:25:21.800519Z DEBUG discv4: removed node node_id=0x7ad7e84a912b98163987cf471e3a9a3cacac4839cf6b6a7fc585329f66e3a5f6d4ff9ae1c7405584286db8f4c65ea2478829fbdf323e515294603eb0906cc3d6
2024-02-20T00:25:21.800520Z DEBUG discv4: removed node node_id=0x51d6d469ee5cd980d70f8e43a8ad88a87a5db18f5af6d8861c3f8dcb9181de8cd20be3a0bd9e504eeef14a4b0a6e61ec32d39a96407c9f48cdab0ea16367bb79
2024-02-20T00:25:21.800522Z DEBUG discv4: removed node node_id=0xaba40e1ae60e82866655c2ab8abc25d7e7c279d1d9f2d6ae960726641478f0943ad3e74d6ce03882b73ae158e3b40354a2238883ebb94a33fb907f992acf8b96
2024-02-20T00:25:21.800523Z DEBUG discv4: removed node node_id=0xa9297932fef5cd39a1d7491403bcf21bdc5ba4b2a363ddf2bb0048e08496d34ad3ba7b4b0371a5dc38cd4b21d2ddbe40c02d698e1ba4e17ec40449cbda354851
2024-02-20T00:25:21.800524Z DEBUG discv4: removed node node_id=0xc30eb79080c8b8e6438fb04d6520a95eb034618efe86bf86c022242dfc9bc70cc0e379d0a44e6636139a319e86fd025a973d601e564d0ce75a7d8f45940c7c88
2024-02-20T00:25:21.800525Z DEBUG discv4: removed node node_id=0x2c319895b213a2883ed02dd3efa2c4c1a3a40a4df5c15791f2ae241f61925beefa90e7449572ee6c037f161fb6700b0b7074432e07d9869f0ca85f42a7fff204
2024-02-20T00:25:21.800526Z DEBUG discv4: removed node node_id=0x062e3f1c673c6b1e49d6f7e46e65304fb1674b01ec60584b368b1eb760a295324aaa63250e0ffb43fad9a7699bd3014c404c8fa67412d9a020f58db4e08e5e07
2024-02-20T00:25:21.800527Z DEBUG discv4: removed node node_id=0x8b602fde18a85ffb1156798e86ca7ab79be08125d27e18157bf72324f54a8cbaea4ff30fe2d5c9cc5270d7cdc78a79181718313a0cc26fb89ebf87fa0fc2d80c
2024-02-20T00:25:21.800529Z DEBUG discv4: removed node node_id=0x493c64b85aaca264095bb9fbf26547943a2697a25ae398c6ffe2c1a1623d4b56d7263f783cc06f6d0a9fdaf67de02d14b3ccd72cd6cf48a64093a5030e8cb428
2024-02-20T00:25:21.800530Z DEBUG discv4: removed node node_id=0xe5c3266c8dfb0f7b0d29f4ab03eb00425c4d4f82b3eb6d81cbe1debba353977fb86c769d1668dea710d4c5be652d3053082b0e2499340e5959b2f6142e403b22
2024-02-20T00:25:21.800531Z DEBUG discv4: removed node node_id=0x951ea48226963df6483caa52cb9a8e38b7324e4741e9d18668b5f26b0166b3e350d4415a7d9e5bbd83e98a5912d1108bdcf28154a57704a4827bb97eab6e5d7c
2024-02-20T00:25:21.800533Z DEBUG discv4: removed node node_id=0x0df666cf7b4fbe8e27a2d1e2bc7c40154ac14d8f0a1f17039cb3cfdce5d51ae23fbf97b3ac37dfc28aeb17333c733adebb1818ed3997a33f29eb507e4ce695aa
2024-02-20T00:25:21.800534Z DEBUG discv4: removed node node_id=0x00a7d0a7ebf7e9196d580caa200f85532d55a0710f5d92e92e447df4ff4dbafc03e45dba9dbe7a23a5f1fccf5366171bae6ffe7084dc53d7b3e79e99824a4464
2024-02-20T00:25:21.800536Z DEBUG discv4: removed node node_id=0x057bb6884b75c8540936bf7a1b253b9dd65903b8286a320f3ee77b7124e87e8ab1baee774a1d5024222ba75218ddf5c575e11dde8457c795984354f950561484
2024-02-20T00:25:21.800538Z DEBUG discv4: removed node node_id=0xd8f1604705df79feb6edac12670720d4730f5f2111421163d925953bf573c7d48017f40f4944c96a7dcbed47bb9d73bdaf2140288f72b78ba8f76c57ca63b5af
2024-02-20T00:25:21.800541Z DEBUG discv4: evicting nodes due to failed lookup num=10
2024-02-20T00:25:21.800542Z DEBUG discv4: removed node node_id=0x04bbf68b2c5f0099a95b1179374911876d6c53b5d2e944fef30193f7972258c306f95da839c7cfaac1040773b8e01a8ebbad847eccca247a2740bd626b5dcee4
2024-02-20T00:25:21.800543Z DEBUG discv4: removed node node_id=0x046cdbbd92cdf4fc6cafb230b814ad1504f0463816ae7e715d54388786ac42ea4c65026404ab84c94a6b946af5aaf2e053369d228e7e0041b5eab5f5b86f5ef3
2024-02-20T00:25:21.800544Z DEBUG discv4: removed node node_id=0x632fa253160789031539ce78dc22bb697e2040f52074a0316a2374da0b9728b444918a9f8061ce1ebb39f4b1206d0f670dd13ba0fa98fd60eee9ca85d77377f6
2024-02-20T00:25:21.800546Z DEBUG discv4: removed node node_id=0xfa04eb41c2a51747516e011c8977e76b4e5b7e4b155018bd0e714f021182299922b3aefc159d480421a16380e2a1486f6070e8073371ff194bd11ad47b2f823a
2024-02-20T00:25:21.800547Z DEBUG discv4: removed node node_id=0x3db40eb7a48264d01c92dfed8df3a275cfe70ae894c32b11c9c9c9a828cec1fac0b871faa2e5b8b0632f606be9912ddddfb8b32bbea5f3626c644b5e82d9072f
2024-02-20T00:25:21.800548Z DEBUG discv4: removed node node_id=0xfd951ae8b8ae61c9d59dd264bf2812243fd9a94c677b1618e29d78ed36ba0d5c7c548faf35d337c8396f15700de8e5aefcd36de56634506627c3165f5d90e116
2024-02-20T00:25:21.800551Z DEBUG discv4: removed node node_id=0xfd35094ed5d1f4011539808f91436e2f52dea5730e83461be2bab99f851e373f8208cd58b5bca1461d0e02f5f606803b3a99c5efbf97aa775703130be8e6c5f0
2024-02-20T00:25:21.800558Z DEBUG discv4: processing failed neighbours num=3
2024-02-20T00:25:21.907663Z DEBUG net::session: failed to receive message err=P2PStreamError(Disconnected(TooManyPeers)) remote_peer_id=0x1e41de28ba40406a32aaac6c990ee535577fb4028bec5a723602bb65c59dcfcf1daf3b94325887ded034bb8fe96ee2e82cc1eb6d350dc06da2ada5078f2245ff
2024-02-20T00:25:21.910605Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 84.188.87.193, tcp_port: 30305, udp_port: 30305, id: 0x59dfca3d7c83438c1894b8d576625c5b465e93c11e9daa4b96dacac1bab5985348591a2ae7b552f76d6bc5a02d192775c286280e065fa2290887c0295e2790ff }
2024-02-20T00:25:21.949961Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 162.55.232.96, tcp_port: 19921, udp_port: 19921, id: 0x7d0e43418142d16a9c777130a14c576e541682b021975c979ebf695fa935a53ee1092189dbc8a396ed2f9d7ba6d43ace60aa85fbd63892b73f94f90d56ede24e }
2024-02-20T00:25:21.954675Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 149.90.122.45, tcp_port: 40303, udp_port: 40303, id: 0x40582cf7301675a68bd360c5fba6bc280ce64d691dedf91221cfe26d59de78eab8284c930bb33d9ec19b07e636a7386d1a3218ff17952702a35ce7e34450da95 }
2024-02-20T00:25:21.971444Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 174.114.21.152, tcp_port: 30303, udp_port: 30303, id: 0x4dbfc7f9e218a309227a503024aa05332ad793e4d8236c3601f08d514309209ff2d357241b58e512aa5b8405a4d72aeb6d3a224d063a27a68c4d4582e78331ce }
2024-02-20T00:25:22.047118Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 141.95.196.16, tcp_port: 50560, udp_port: 30311, id: 0x3d4370f0d8f6936e1b0ab6197266ddf846e65b0382b1593d3202f9d97db2dfb7dcd0c042563e7b08e0733ff6e2124cb6daae26649b579e8a740fce283b4037df }
2024-02-20T00:25:22.104044Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 185.8.104.188, tcp_port: 30303, udp_port: 30303, id: 0x68cad62102290ff30b159db80a3f815073b68108c295defc67f25d9e146b8b136aa3d51626027c1ac7290fc013318db0d994cc83910a4f77b0a0b40cb1eafa0c }
2024-02-20T00:25:22.821303Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m185.8.104.188:30303}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-20T00:25:22.826433Z DEBUG net::session: failed to receive message err=P2PStreamError(Disconnected(TooManyPeers)) remote_peer_id=0xa9e93522cb766590952ffd6e4b5bb14cfb179b5cbdc874179f27e31688da99cdcd450d6b1357933b07ae115fd4a3b3f55d6d269caa852d21b13077c6e82fb806
2024-02-20T00:25:22.905547Z DEBUG connection{�[3mremote_addr�[0m�[2m=�[0m192.168.208.6:47412 �[3mconn_id�[0m�[2m=�[0m44}: jsonrpsee_server::server: Accepting new connection 3/500
2024-02-20T00:25:23.006490Z DEBUG disc::dns: No dns entry domain=all.mainnet.ethdisco.net hash="EDPE5DD56NFUHTJIFDUMPB"
2024-02-20T00:25:23.083441Z DEBUG net::session: failed to receive message err=P2PStreamError(Disconnected(TooManyPeers)) remote_peer_id=0xdf7ac5ee488b74924bdb861350f174764014ded895332e61d7d8f7672d5b8421e73549bb9b71f0ec2509d179eceac9f5673cb3f69bc5ae2008da3633b17b22bd
2024-02-20T00:25:23.529566Z DEBUG discv4: removed node node_id=0x790a584fddac4838d58be8365824ba694b9e54f6976f533e4343a4e90d75d265b15742d805bd5c4871bdc437119d65fd2fb2d5103a051fbdb3682b99907d6c28
2024-02-20T00:25:23.654560Z DEBUG net::session: failed to receive message err=P2PStreamError(Disconnected(TooManyPeers)) remote_peer_id=0x9101e7279047c6db938c7f17d0435f075bcc3e03b0f5e12a3a1456c6f0ebf10b246c564e8019a2e60159d49474067446de174cb34a123028874894bb47a35705
2024-02-20T00:25:26.008887Z DEBUG disc::dns: No dns entry domain=all.mainnet.ethdisco.net hash="C53XWHUB3CPMV7RMCGYK2C"
2024-02-20T00:25:27.088899Z  INFO reth_node_core::events::node: Forkchoice updated head_block_hash=0xea32e47e30b3150b48d9a359fdd8d4759b52e5f39aa21423ecf06d123eebbc6f safe_block_hash=0xb6295ff4a08fdcb8e699dad078035a08ef4d72533905798d40303ba619d64003 finalized_block_hash=0x7cd519fb7a2b0e9055a41c44231c916852143af9a691843954f549df87b755e7 status=Syncing
2024-02-20T00:25:29.019971Z DEBUG disc::dns: No dns entry domain=all.mainnet.ethdisco.net hash="SFKJI4UG2Q5NVQJI4ZP2F7"
2024-02-20T00:25:31.002361Z DEBUG connection{�[3mremote_addr�[0m�[2m=�[0m192.168.208.1:53652 �[3mconn_id�[0m�[2m=�[0m45}: jsonrpsee_server::server: Accepting new connection 3/500
2024-02-20T00:25:32.021571Z DEBUG disc::dns: No dns entry domain=all.mainnet.ethdisco.net hash="XDMAZ2TWFP52FGVOG53DKH"
2024-02-20T00:25:34.802461Z  INFO reth::cli: Status connected_peers=8 freelist=1552658 stage=Execution checkpoint=19259458 target=19262463 stage_progress=None
2024-02-20T00:25:35.023394Z DEBUG disc::dns: No dns entry domain=all.mainnet.ethdisco.net hash="TCXMB4OPPZMJYCGRP6QBH6"
2024-02-20T00:25:37.301867Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0xb8bd…adf1" hash=0x666b3defd5c3c8ba3f1300ca36e05ca4e1a408632f2e0bf6dbc2055feb7b9f44 size=180 previously_seen_size=182 client_version=Geth/v1.11.5-stable-a38f4108/linux-amd64/go1.20.2
2024-02-20T00:25:37.301888Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0xb8bd…adf1" hash=0x1265fc62048447f37b626d44bff3b3a49eb44d546dd575fc8a1bda11e00d6c69 size=180 previously_seen_size=182 client_version=Geth/v1.11.5-stable-a38f4108/linux-amd64/go1.20.2
2024-02-20T00:25:37.301893Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0xb8bd…adf1" hash=0x7ea6503fa2a5a69f835c007f332440606f439522070d0a11cd3f1593084773d6 size=178 previously_seen_size=180 client_version=Geth/v1.11.5-stable-a38f4108/linux-amd64/go1.20.2

@krakovia-evm
Copy link
Author

tried for the third time to unwind.
The node is synced now but i don't get any useful log
it's filled with those

2024-02-20T15:56:58.733055Z DEBUG discv4: evicting nodes due to failed pong num=0
2024-02-20T15:56:58.733069Z DEBUG discv4: evicting nodes due to failed lookup num=0
2024-02-20T15:56:58.733072Z DEBUG discv4: processing failed neighbours num=0
2024-02-20T15:56:59.630871Z DEBUG connection{�[3mremote_addr�[0m�[2m=�[0m172.18.0.1:56234 �[3mconn_id�[0m�[2m=�[0m123}: jsonrpsee_server::server: Accepting new connection 3/500
2024-02-20T15:57:02.388093Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m76.95.4.97:30303}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-20T15:57:03.960120Z DEBUG net::session: timed out outgoing request id=408 remote_peer_id=0x71b4bc0970a9dcf39e8524b04bbcaebb412b6bb3e5c4364a9638d04d7145cfc5a4cf363b3c86fe1afaa4cbf99e201880b07b31602e369b931d33806ea9c29d8b
2024-02-20T15:57:05.263346Z  INFO reth_node_core::events::node: Forkchoice updated head_block_hash=0x0227668b196787e641c89770237a83b08fe821c3879dae4d56db34b0c440e686 safe_block_hash=0xd03673555f5747db442ce90b1905da5ca1a9c928885423065b5b146c1c12ec0d finalized_block_hash=0x9ae8069c627d8773fb52bdae9acdc6ee4906c7521da1575ecfa740a1f36d33bf status=Syncing
2024-02-20T15:57:06.739576Z  INFO reth::cli: Status connected_peers=20 freelist=6672698 stage=Execution checkpoint=19259458 target=19267130 stage_progress=None
2024-02-20T15:57:06.780881Z DEBUG net::session: timed out outgoing request id=409 remote_peer_id=0x71b4bc0970a9dcf39e8524b04bbcaebb412b6bb3e5c4364a9638d04d7145cfc5a4cf363b3c86fe1afaa4cbf99e201880b07b31602e369b931d33806ea9c29d8b
2024-02-20T15:57:08.891196Z DEBUG net::session: failed to receive message err=P2PStreamError(Disconnected(TooManyPeers)) remote_peer_id=0x713ac7ab8c1086495bdd42354e0c6240893845a7f586cac2e8acc2700e1ac55a596cb886c8574961d5c865c4d665c9a3bb34ab4b64076178e4d00c0375f50b2a
2024-02-20T15:57:08.891404Z DEBUG net::session: failed to receive message err=P2PStreamError(Disconnected(TooManyPeers)) remote_peer_id=0xf930e4054447c5062e49e1fef45fc10e0a9fbe2d8d150a3ef2008146c93a338f08abaa12fd93658f1cb60e7726d51c48aa60e3355c301416d89517bdd44e2ae4
2024-02-20T15:57:08.911921Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m77.109.174.30:30303}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-20T15:57:08.946891Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m176.9.35.94:30303}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-20T15:57:09.132453Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m107.134.180.138:30303}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-20T15:57:09.901224Z DEBUG connection{�[3mremote_addr�[0m�[2m=�[0m192.168.240.6:39110 �[3mconn_id�[0m�[2m=�[0m124}: jsonrpsee_server::server: Accepting new connection 3/500
2024-02-20T15:57:11.001190Z DEBUG net::session: timed out outgoing request id=410 remote_peer_id=0x71b4bc0970a9dcf39e8524b04bbcaebb412b6bb3e5c4364a9638d04d7145cfc5a4cf363b3c86fe1afaa4cbf99e201880b07b31602e369b931d33806ea9c29d8b
2024-02-20T15:57:13.834985Z DEBUG connection{�[3mremote_addr�[0m�[2m=�[0m192.168.240.3:51018 �[3mconn_id�[0m�[2m=�[0m4}: jsonrpsee_server::server: Accepting new connection 3/500
2024-02-20T15:57:14.637599Z DEBUG connection{�[3mremote_addr�[0m�[2m=�[0m172.18.0.1:56928 �[3mconn_id�[0m�[2m=�[0m125}: jsonrpsee_server::server: Accepting new connection 3/500
2024-02-20T15:57:18.732645Z DEBUG discv4: evicting nodes due to failed pong num=0
2024-02-20T15:57:18.732653Z DEBUG discv4: evicting nodes due to failed lookup num=0
2024-02-20T15:57:18.732657Z DEBUG discv4: processing failed neighbours num=0
2024-02-20T15:57:18.836894Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 213.157.24.66, tcp_port: 30303, udp_port: 30303, id: 0x9f3d81b5b3933e437d4f1112e4fd2edc403d339b0f2b71a9e16e57562bd59bc924f99e1a7602c0660206ff2e7de9c3a80b1389d2b2f4da2ee30a7e8e624eee5a }
2024-02-20T15:57:19.028038Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 198.185.218.175, tcp_port: 30309, udp_port: 30309, id: 0x95d4b03393500659626fd1268182dea1b50494c5ab71b45752999df31a46c837b4a58845fcc10c5e6c4b5b627d91aa0bfd92f67a0b33f4cfdb0491f63b74a8cc }
2024-02-20T15:57:19.042066Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 54.254.46.0, tcp_port: 30303, udp_port: 30303, id: 0x0427cb60d6e890f5371a7477b5ca311b4ef859492941580bc23472bd6a901619208340afcac02b4c002639e73d3dc4939d456205895cf0c910e52619db06ddb0 }
2024-02-20T15:57:19.091397Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 124.191.150.243, tcp_port: 30303, udp_port: 30303, id: 0xe6d95ffde0f2b9400cc7c63daca1eb77fff0328dc0fed464cfb8b8f785228e76609df536e5fcb8ef50077f9a376dd3da0c6357a9344a4029cd5f7b67162ee167 }
2024-02-20T15:57:20.263240Z  INFO reth_node_core::events::node: Forkchoice updated head_block_hash=0xffa8853b374334c16547589837bdac7992c6d067e0ba770ece1bfcd94f590ef2 safe_block_hash=0xe981cb82eac7bcacbe1fa3d498edd15750ef8a1ecad496076652078cfcee0bb5 finalized_block_hash=0x0227668b196787e641c89770237a83b08fe821c3879dae4d56db34b0c440e686 status=Syncing
2024-02-20T15:57:22.314410Z DEBUG net::eth-wire: invalid tx size in eth68 announcement ty=0 size=0 hash=0xcbec0c668ad4d67c80c11fe4165c6407c8ef1731ce7d294ed1951edf04ec353b strict_min_encoded_tx_length=1 network=EthAnnouncementFilter
2024-02-20T15:57:22.314423Z DEBUG net::eth-wire: invalid tx size in eth68 announcement ty=0 size=0 hash=0x562ff6b4ede41dc56d17fbe6db9438f98152e6b2eb2ca5814826adec7d3ccc1e strict_min_encoded_tx_length=1 network=EthAnnouncementFilter
2024-02-20T15:57:22.314425Z DEBUG net::eth-wire: invalid tx size in eth68 announcement ty=0 size=0 hash=0xa8e64f7e6614e99dcb580d86c7427fdac42f80b84daac17f7778b16f5cb1cfe2 strict_min_encoded_tx_length=1 network=EthAnnouncementFilter
2024-02-20T15:57:22.314426Z DEBUG net::eth-wire: invalid tx size in eth68 announcement ty=0 size=0 hash=0xb81b0f8731bf4c57f6470085169d1051a5a680db09022fc3b730e293ed5b1980 strict_min_encoded_tx_length=1 network=EthAnnouncementFilter
2024-02-20T15:57:22.314427Z DEBUG net::eth-wire: invalid tx size in eth68 announcement ty=0 size=0 hash=0x227b991cde98dd368a13f782eb1fad4bbf6ae75d3aa966089a3bfbd43dcf59e1 strict_min_encoded_tx_length=1 network=EthAnnouncementFilter
2024-02-20T15:57:22.314428Z DEBUG net::eth-wire: invalid tx size in eth68 announcement ty=0 size=0 hash=0x27e4f355bbd8a0e8c41cb98fd2a984298e8344de857f788af72c3ff355cb7dc2 strict_min_encoded_tx_length=1 network=EthAnnouncementFilter
2024-02-20T15:57:22.314429Z DEBUG net::eth-wire: invalid tx size in eth68 announcement ty=0 size=0 hash=0x53b2623bf3fb991bf904e562886542c71a04c8c6afa36f065a732f7fc63ab4c0 strict_min_encoded_tx_length=1 network=EthAnnouncementFilter

@krakovia-evm
Copy link
Author

krakovia-evm commented Feb 23, 2024 •
edited
Loading

i did "reth stage drop merkle" and unwind back to a "good" block.
Seems it partially fixed the issue but the logs are still filled with that issue
Also the sync is "dancing", not stable

2024-02-23T13:21:46.501261Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0x42b5…06f6" hash=0xa11e4e64d96c175b30d40a350b0644bccbdb2bb7a618705a2026db06f54ef504 size=763 previously_seen_size=760 client_version=erigon/v2.47.0-stable-c5a5df55/linux-amd64/go1.19.5
2024-02-23T13:21:46.508934Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0xa49d…a1b8" hash=0xa11e4e64d96c175b30d40a350b0644bccbdb2bb7a618705a2026db06f54ef504 size=760 previously_seen_size=763 client_version=Geth/v1.13.8-stable-b20b4a71/linux-amd64/go1.21.5
2024-02-23T13:21:46.535903Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0xabec…a121" hash=0xd3b26f9c4264e77ca921510e8d6a6b650d7110ef13f6fb16960558a1d8f451a2 size=184 previously_seen_size=182 client_version=erigon/v2.48.0/linux-amd64/go1.20
2024-02-23T13:21:46.536754Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0x5684…6d5a" hash=0xd3b26f9c4264e77ca921510e8d6a6b650d7110ef13f6fb16960558a1d8f451a2 size=182 previously_seen_size=184 client_version=Geth/v1.11.5-stable/linux-amd64/go1.21.0
2024-02-23T13:21:47.092099Z DEBUG blockchain_tree: Committing new canonical chain: [(19290551, 0x0b530dd6f4b7db753745fbb20a55198793653748f9f1950ee21ef1dfb1fdf876), (19290552, 0xeda6b6ffe924325b8f76a16725a592a0b4a52f596fa3f6b7e641eaa0b449f82a), (19290553, 0x8146db6e960794434675b79d1d04754bbe1164446978a9245ef03a1fa5fc6b27)]
2024-02-23T13:21:47.101080Z DEBUG blockchain_tree: Using cached trie updates blocks=[(19290551, 0x0b530dd6f4b7db753745fbb20a55198793653748f9f1950ee21ef1dfb1fdf876), (19290552, 0xeda6b6ffe924325b8f76a16725a592a0b4a52f596fa3f6b7e641eaa0b449f82a), (19290553, 0x8146db6e960794434675b79d1d04754bbe1164446978a9245ef03a1fa5fc6b27)]
2024-02-23T13:21:47.150455Z DEBUG providers::db: Inserted block block_number=19290551 actions=[(InsertCanonicalHeaders, 82.143µs), (InsertHeaders, 51.315µs), (InsertHeaderNumbers, 34.967µs), (GetParentTD, 2.514µs), (InsertHeaderTD, 39.239µs), (GetNextTxNum, 12.525µs), (InsertTxSenders, 0ns), (InsertTransactions, 7.396902ms), (InsertTxHashNumbers, 41.42214ms), (InsertBlockWithdrawals, 36.298µs), (InsertBlockBodyIndices, 30.16µs), (InsertTransactionBlock, 30.263µs)]
2024-02-23T13:21:47.227285Z DEBUG providers::db: Inserted block block_number=19290552 actions=[(InsertCanonicalHeaders, 4.24µs), (InsertHeaders, 9.338µs), (InsertHeaderNumbers, 25.791µs), (GetParentTD, 1.958µs), (InsertHeaderTD, 4.553µs), (GetNextTxNum, 8.635µs), (InsertTxSenders, 0ns), (InsertTransactions, 24.62433ms), (InsertTxHashNumbers, 52.038471ms), (InsertBlockWithdrawals, 8.788µs), (InsertBlockBodyIndices, 4.255µs), (InsertTransactionBlock, 3.726µs)]
2024-02-23T13:21:47.227493Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0xabec…a121" hash=0x43c64b53a0d3d73b60d219de6d596252bbbd796b23a79c27a819ea86107d2846 size=190 previously_seen_size=188 client_version=erigon/v2.48.0/linux-amd64/go1.20
2024-02-23T13:21:47.228123Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0xfd55…6bf1" hash=0x43c64b53a0d3d73b60d219de6d596252bbbd796b23a79c27a819ea86107d2846 size=188 previously_seen_size=190 client_version=Geth/v1.11.5-stable/linux-amd64/go1.21.0
2024-02-23T13:21:47.237989Z DEBUG providers::db: Inserted block block_number=19290553 actions=[(InsertCanonicalHeaders, 3.006µs), (InsertHeaders, 6.692µs), (InsertHeaderNumbers, 18.859µs), (GetParentTD, 1.828µs), (InsertHeaderTD, 4.299µs), (GetNextTxNum, 6.885µs), (InsertTxSenders, 0ns), (InsertTransactions, 378.071µs), (InsertTxHashNumbers, 10.21963ms), (InsertBlockWithdrawals, 12.993µs), (InsertBlockBodyIndices, 3.959µs), (InsertTransactionBlock, 3.795µs)]
2024-02-23T13:21:47.382936Z DEBUG providers::db: Appended blocks range=19290551..=19290553 actions=[(InsertBlock, 49.248051ms), (InsertBlock, 76.823451ms), (InsertBlock, 10.702087ms), (InsertState, 22.723005ms), (InsertHashes, 90.56479ms), (InsertHistoryIndices, 31.618261ms), (UpdatePipelineStages, 19.184µs)]
2024-02-23T13:21:47.745484Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 35.268µs), (FindCanonicalHeader, 67.719µs), (SplitChain, 4.901µs), (SplitChainForks, 2.152µs), (MergeAllChains, 2.29µs), (UpdateCanonicalIndex, 9.999µs), (RetrieveStateTrieUpdates, 9.011041ms), (CommitCanonicalChainToDatabase, 644.365349ms)]
2024-02-23T13:21:47.745554Z DEBUG consensus::engine: Canonicalized new head hash=0x8146db6e960794434675b79d1d04754bbe1164446978a9245ef03a1fa5fc6b27 number=19290553
2024-02-23T13:21:47.745775Z  INFO reth_node_core::events::node: Canonical chain committed number=19290553 hash=0x8146db6e960794434675b79d1d04754bbe1164446978a9245ef03a1fa5fc6b27 elapsed=653.549687ms
2024-02-23T13:21:48.189477Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0x669f…1c6f" hash=0x3ca6d4d17f3918338af0497d738c943224cd07048dff9d0a78ae12a58651cbf4 size=183 previously_seen_size=181 client_version=erigon/v2.48.1/linux-amd64/go1.20.2
2024-02-23T13:21:48.192019Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0xc6af…9264" hash=0x3ca6d4d17f3918338af0497d738c943224cd07048dff9d0a78ae12a58651cbf4 size=181 previously_seen_size=183 client_version=Geth/v1.13.11-unstable-6e235c08-20240112/linux-amd64/go1.20.12
2024-02-23T13:21:48.213111Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0x42b5…06f6" hash=0x3ca6d4d17f3918338af0497d738c943224cd07048dff9d0a78ae12a58651cbf4 size=183 previously_seen_size=181 client_version=erigon/v2.47.0-stable-c5a5df55/linux-amd64/go1.19.5
2024-02-23T13:21:48.213993Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0x2160…ca68" hash=0x3ca6d4d17f3918338af0497d738c943224cd07048dff9d0a78ae12a58651cbf4 size=181 previously_seen_size=183 client_version=Geth/v1.13.12-stable/linux-amd64/go1.22.0
2024-02-23T13:21:48.471881Z DEBUG blockchain_tree: Appending block to canonical chain head=0x3b72a221c821d19a91c40d72546273a76e02bd5839d63890d7e7d8491966a206 parent=(19290553, 0x8146db6e960794434675b79d1d04754bbe1164446978a9245ef03a1fa5fc6b27)
2024-02-23T13:21:48.528555Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0xabec…a121" hash=0x1fed8b227e5ba778fd307ac76e2e635be6fd8e1454c94bc94662d7babb9943cb size=185 previously_seen_size=183 client_version=erigon/v2.48.0/linux-amd64/go1.20
2024-02-23T13:21:49.162931Z DEBUG blockchain_tree::chain: Validated state root number=19290554 hash=0x3b72a221c821d19a91c40d72546273a76e02bd5839d63890d7e7d8491966a206 elapsed=579.721531ms
2024-02-23T13:21:49.163823Z  INFO reth_node_core::events::node: Block added to canonical chain number=19290554 hash=0x3b72a221c821d19a91c40d72546273a76e02bd5839d63890d7e7d8491966a206 peers=113 txs=398 mgas=29.990 full=100.0% base_fee=34.68gwei blobs=0 excess_blobs=0 elapsed=695.574449ms
2024-02-23T13:21:49.318967Z DEBUG blockchain_tree: Committing new canonical chain: [(19290554, 0x3b72a221c821d19a91c40d72546273a76e02bd5839d63890d7e7d8491966a206)]
2024-02-23T13:21:49.323750Z DEBUG blockchain_tree: Using cached trie updates blocks=[(19290554, 0x3b72a221c821d19a91c40d72546273a76e02bd5839d63890d7e7d8491966a206)]
2024-02-23T13:21:49.405480Z DEBUG providers::db: Inserted block block_number=19290554 actions=[(InsertCanonicalHeaders, 40.111µs), (InsertHeaders, 30.913µs), (InsertHeaderNumbers, 16.302µs), (GetParentTD, 962ns), (InsertHeaderTD, 13.758µs), (GetNextTxNum, 6.65µs), (InsertTxSenders, 0ns), (InsertTransactions, 6.617275ms), (InsertTxHashNumbers, 74.703616ms), (InsertBlockWithdrawals, 65.504µs), (InsertBlockBodyIndices, 20.44µs), (InsertTransactionBlock, 17.844µs)]
2024-02-23T13:21:49.562487Z DEBUG net::tx: peer announced a different size for tx, this is especially worrying if one size is much bigger... peer_id="0xbc90…4bfd" hash=0x6d92bcbfa15e69636913e2cbda0fadb87780ac552fdaf0787e8a922fde201ccf size=121 previously_seen_size=123 client_version=Geth/v1.11.5-stable-c5985be4/linux-amd64/go1.20.1
2024-02-23T13:21:49.566845Z DEBUG providers::db: Appended blocks range=19290554..=19290554 actions=[(InsertBlock, 81.677381ms), (InsertState, 25.143544ms), (InsertHashes, 98.856042ms), (InsertHistoryIndices, 37.307372ms), (UpdatePipelineStages, 18.76µs)]
2024-02-23T13:21:49.863774Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 52.564µs), (FindCanonicalHeader, 28.588µs), (SplitChain, 2.314µs), (SplitChainForks, 1.456µs), (MergeAllChains, 1.076µs), (UpdateCanonicalIndex, 5.927µs), (RetrieveStateTrieUpdates, 4.797835ms), (CommitCanonicalChainToDatabase, 540.00001ms)]
2024-02-23T13:21:49.863855Z DEBUG consensus::engine: Canonicalized new head hash=0x3b72a221c821d19a91c40d72546273a76e02bd5839d63890d7e7d8491966a206 number=19290554
2024-02-23T13:21:49.864019Z  INFO reth_node_core::events::node: Canonical chain committed number=19290554 hash=0x3b72a221c821d19a91c40d72546273a76e02bd5839d63890d7e7d8491966a206 elapsed=544.982591ms
2024-02-23T13:21:52.756682Z DEBUG discv4: evicting nodes due to failed pong num=16
2024-02-23T13:21:52.756693Z DEBUG discv4: removed node node_id=0xafbef9d6d1e52c2e5e8a27c7b69516337f7b725174b60703cbf7121ad3e9b709f8d59b6a2532b80d503c85ae695c83cf4175e1162de778b0b24dfe9543d46431
2024-02-23T13:21:52.756696Z DEBUG discv4: removed node node_id=0x9ac1bb12ac2b3114583213d0d11cad0d4025c3e48a01d26565aab6d7fad4cee7ac46ac8d5bd5f6a6c09ec7a9851bb6aa9780af0bd5f67212547a2e8aaf7c1a1e
2024-02-23T13:21:52.756698Z DEBUG discv4: removed node node_id=0x206c5fadbc8394be71b1cf2d55b417fbd02f0b2760c5742c45041a7231b66c9d2a99f74f38d4912b087d9800cb643bb726f94672fa9d11a94de098f11c80240f
2024-02-23T13:21:52.756699Z DEBUG discv4: removed node node_id=0x4f0561c641b1e97d8016c73693ceddc2d975372854b4d5fa8a2a0b050d857c2e882409985f1d20e1f33a5a013fba8b94caa0a0d57ef604ef7acefc8faeb1596c
2024-02-23T13:21:52.756700Z DEBUG discv4: removed node node_id=0x999c6d6c214a983719ca33154406b554a0f03e6d91d41528ea53d0a3c57da6f7f5d14b98b42177717f44048dfa532dffdb07820ce4fd6b183297fc7cffe26cb6
2024-02-23T13:21:52.756701Z DEBUG discv4: removed node node_id=0xf085918b75d2284cd53e1b90556762fa23f5764c5dba62555b20f984eda1385e4920f6c661f4a1e9c2ae51e7ba3ddd85db755c814426a3fa6a59baaa744a94df
2024-02-23T13:21:52.756703Z DEBUG discv4: removed node node_id=0xa2c5285207d434a8582dcc02d9940d6ef49b971760c7b074b025223d1f545dcb903aa305aa7343eab170a7c1b4918f7455c825043af6f37f21c2ea1c6f6c088a

Also i've spotted an unknown pattern in this issue.
image

probably this?
image
that's a subscription to newHeads, the timer near the clock is the difference in time between machine's time & block.timestamp

image

@mattsse
Copy link
Collaborator

mattsse commented Feb 23, 2024 •
edited
Loading

@emhane all those logs should be trace!

@mattsse mattsse mentioned this issue Feb 23, 2024
Closed
@krakovia-evm
Copy link
Author

seems i'm still in sync, so dropping the merkle table indeed helped out, but still the node is not efficent.
While the logs are still heavly spammed (since last post, i now got 4 files of 209mb 🙆‍♂️), i think this is what is making the performance worst
image

@krakovia-evm
Copy link
Author

krakovia-evm commented Feb 25, 2024 •
edited
Loading

oof, seems i've reached another dead point.
Sync lost and both reth + lighthouse are standby.
RETH logs

2024-02-25T03:37:20.734174Z DEBUG blockchain_tree: Appending block to canonical chain head=0x0783543e234eb2c87f88fc17bdb4a89edd6f827d968765e4472e85e3effefd72 parent=(19301936, 0x4ac1a571f6793d9f1ae4d465da9599a367aa0a4cc0324cea26171bacc91969f4)
2024-02-25T03:37:20.838967Z  WARN consensus::engine: Error while processing payload error=InsertBlockError { error: Consensus(BodyStateRootDiff(GotExpected { got: 0xa395dee2898afaff56f579fb7e76d2419746e064d80fa121cc26c0a5c133fb09, expected: 0xa19729e38a82a7acb473ced6f5237544ea368b7e371d2c97ec6fab98e5fe3632 })), hash: 0x0783543e234eb2c87f88fc17bdb4a89edd6f827d968765e4472e85e3effefd72, number: 19301937, parent_hash: 0x4ac1a571f6793d9f1ae4d465da9599a367aa0a4cc0324cea26171bacc91969f4, num_txs: 150, .. }
2024-02-25T03:37:20.838977Z  WARN consensus::engine: Invalid block error on new payload invalid_hash=0x0783543e234eb2c87f88fc17bdb4a89edd6f827d968765e4472e85e3effefd72 invalid_number=19301937 error=Consensus(BodyStateRootDiff(GotExpected { got: 0xa395dee2898afaff56f579fb7e76d2419746e064d80fa121cc26c0a5c133fb09, expected: 0xa19729e38a82a7acb473ced6f5237544ea368b7e371d2c97ec6fab98e5fe3632 }))

Lighthouse
image

edit: seems i'm back at the starter point 🙆‍♂️
after unwind 100 blocks

2024-02-25T04:59:20.187131Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 185.245.182.236, tcp_port: 30303, udp_port: 30303, id: 0x7a9758ab9df656bebcfba4617c0d5e5eeaadd3d845b5c21fb5ada93fa781ffaf67e57ebb14e16cfe867a3a4075a21baba3349e8a744acd41caeb527f41a8b162 }
2024-02-25T04:59:20.193968Z  INFO Unwinding{�[3mstage�[0m�[2m=�[0mAccountHashing}: sync::pipeline: Stage unwound stage=AccountHashing unwind_to=19301914 progress=19301914 done=true
2024-02-25T04:59:20.205042Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 3.250.123.67, tcp_port: 30303, udp_port: 30303, id: 0x52967da0d2978e788c766be335dbaa8578b21957b8f05f5bdf919d472cfa05c9f8b9f589c540887c5be44661c7b4fbbbc9b1e9de7c5fb5a45aa1a0a098dbae67 }
2024-02-25T04:59:20.207597Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 77.20.80.60, tcp_port: 30303, udp_port: 30303, id: 0x3565689657f1f918ef34d4485bd8e1a1586b2d5d9487184f4fed3c672225bcf6256b5b0f245b3c8f9a0962a842ffb11149286ee2df3482a68f7c54981b5d8acd }
2024-02-25T04:59:20.223168Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 18.190.47.41, tcp_port: 30303, udp_port: 30303, id: 0x60042eb84d366e57e564aa6061928399031c5b21b307f6e3b7496ad644c95a457dbdee57d541f538a201c2d593e0208ce94c764d64112788fd151134fd7b6a9c }
2024-02-25T04:59:20.223228Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 18.221.119.208, tcp_port: 30303, udp_port: 30303, id: 0xb106fcb82323e080c56677353e44f04a078a0aa14b30e11ad249b511a9c9e4c69c2772088b831822b12bf00629a74ee511267c385eafea562b4473479d158b99 }
2024-02-25T04:59:20.282766Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 38.34.162.100, tcp_port: 30303, udp_port: 30303, id: 0x3508bf3d7451f6655f563193a13959fe2229bcd1e99725b0edba7d459b58d09407812eb356dd07df0df24121e862223defd43747c4f2ea181cb1b1d622d87fb8 }
2024-02-25T04:59:20.317078Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 73.32.75.111, tcp_port: 30303, udp_port: 30303, id: 0x476293d995a6d84a3a5de4cf03cdfc3e75b05c2aafb9e27a89b77c649cd69f859f8da629d677dcac9ccc9dc6a46b2ec64c7807235968c979b0184fc2abd16c7e }
2024-02-25T04:59:20.344628Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 96.48.54.147, tcp_port: 30303, udp_port: 30303, id: 0xd9764eee75834a18a132bbc07561afb5db21207e9e17818c615e1f490327ea1b845357923e80c6347ee1935b0a9839a19c85f1853802d726c79adf08cbcd0718 }
2024-02-25T04:59:20.365314Z DEBUG discv4: added after successful endpoint proof record=NodeRecord { address: 184.22.17.215, tcp_port: 30304, udp_port: 30304, id: 0x0ed82414e2cdc1914f7b27be055d09d620c6158ab678f08005514cc591d4742140a9ec425d64731e91b8b2be32a5dce89c15d2ab71dae4381dad4f7fddbd21ac }
2024-02-25T04:59:21.056894Z DEBUG disc::dns: No dns entry domain=all.mainnet.ethdisco.net hash="Z44QEJEURPYVQUNR76TLVD"
2024-02-25T04:59:21.307186Z DEBUG Unwinding{�[3mstage�[0m�[2m=�[0mMerkleUnwind}: sync::pipeline: Starting unwind from=19302264 to=19301914 bad_block=Some(19302264)
2024-02-25T04:59:21.307211Z DEBUG Unwinding{�[3mstage�[0m�[2m=�[0mMerkleUnwind}: trie::loader: incremental state root range=19301915..=19302264
2024-02-25T04:59:24.060968Z DEBUG disc::dns: No dns entry domain=all.mainnet.ethdisco.net hash="HCQHUVZFZPTVGEXVYF4SRB"
2024-02-25T04:59:24.507030Z  INFO reth_node_core::events::node: Forkchoice updated head_block_hash=0x6081a5ea877a100e0b4974ff419480866e029ff2472ba96bf31e436922ff80f2 safe_block_hash=0xca6d2199c2425c311bb15aecbc6bed1ab608df37ddc3bb053ab86c4a8c01eec9 finalized_block_hash=0x54ee7cfb2a4c5f7429d0be9f680bc3f91da95118f5873d6a511190fa27bc8b8f status=Syncing
2024-02-25T04:59:25.007456Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m65.108.4.248:30303}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-25T04:59:25.011182Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m95.216.219.243:30303}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-25T04:59:25.074177Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m176.103.222.44:30303}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-25T04:59:25.084385Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m93.193.150.168:30404}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-25T04:59:25.096454Z DEBUG discv4: removed node node_id=0xc9c3ece6b47ad54f8e6c5fd28927149f2fca3ce4077c76e23589de5f98738ef8c9813b4e71c22bd52a7f5686d0fdebd58a54f37b7b74f5eeda8ad63f12d3699a
2024-02-25T04:59:25.108370Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m195.201.12.125:30303}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-25T04:59:25.110313Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m195.201.71.201:30303}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-25T04:59:25.141997Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m178.182.239.4:30404}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-25T04:59:25.156290Z DEBUG discv4: removed node node_id=0x9039ff7117ee73d04ed95e18d791144fc95b59ce3841cf20cce2b325c033e208ac9f9a207c0a76df579ba9f06545203a4658aaf601496a1caf43abd3a13a8d3d
2024-02-25T04:59:25.199367Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m88.5.216.75:31404}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-25T04:59:25.219819Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m162.246.17.18:31303}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-25T04:59:25.283410Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m38.80.96.7:30303}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-25T04:59:25.353718Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m90.215.208.218:30404}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-25T04:59:25.367154Z DEBUG discv4: removed node node_id=0xbbb42216a33cc17f24ccffcb886765a5dccb98cb33ee75bc42c1171a0433d25357b87a6e574ef6677230f0fc782409a1c6c93485a31fa4b69b63c52c8a8ae0d4
2024-02-25T04:59:25.382873Z DEBUG start_pending_outbound_session{�[3mremote_addr�[0m�[2m=�[0m96.48.54.147:30303}: reth_eth_wire::p2pstream: error decompressing p2p message err=HeaderMismatch { expected_len: 577, got_len: 0 } msg=c104
2024-02-25T04:59:26.086861Z  WARN Unwinding{�[3mstage�[0m�[2m=�[0mMerkleUnwind}: sync::stages::merkle: Failed to verify block state root target_block=19301914 got=0x8c3d14f4a800ce86f98b94b29324f3a06922f7537f907fdae0892c8e6ad9aa21 expected=SealedHeader { header: Header { parent_hash: 0x31cb9422624b426b0ad8f9a4315d06ccaa570a4fb9e12f25c569b50fdc697648, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x95222290dd7278aa3ddd389cc1e1d165cc4bafe5, state_root: 0x9bcb9e02a75dee9048eb314fbf7021b7005b226d5f26be857255e548f6bfd922, transactions_root: 0x03790cfb2ea20a3d0671167e7f340d0b48e1051a79718d638d717d9a72d634c9, receipts_root: 0x3c37fba3fa7f1acaee50aa5ef17aadedd6fa28cbd1ab3b4cba4fcce26d7144f4, withdrawals_root: Some(0x49b7b652449f5541e6efe72a8d1506b8c59708a43748f835adf43c5d0eae5b68), logs_bloom: 0x00200403d002104a505806e09c1100a1141183214444111508a6091c7079023099aa0903844840c803923642000411b48a02d0408ac0282100445801902482206814850845200e6c4ca76459042403a4588904902c4c4c5211421c25c8228811d0006c012368a143600490440002e8c13d308c000021c4004140049485081c401a2016100880080104950040035684a06000240105010a1d442408615a38002002e0805010063a01620c44c099000f0002cc8649f01141a0a8118c74198001d00142188228142c0810880a35614e40918452009a060088506800434a4080f00260f1a12d8804a390120461c128400021812492384e102468104a955010021044, difficulty: 0x0_U256, number: 19301914, gas_limit: 30000000, gas_used: 14615597, timestamp: 1708831703, mix_hash: 0x610843b5930e292ba2e66f4347cbc736d11218ae5b31858d687ecc90e8f332b8, nonce: 0, base_fee_per_gas: Some(21566619381), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x6265617665726275696c642e6f7267 }, hash: 0xe888a41ec6a5275a93b494aa6960f3d832b263ae746093e305b6c8dfdfc5c61f }
2024-02-25T04:59:26.155239Z ERROR reth::cli: shutting down due to error
2024-02-25T04:59:28.074789Z  INFO reth::cli: reth 0.1.0-alpha.19 (99d1d8daf) starting

now on rewind

Error: Transaction error on unwind: unwind merkle trie root mismatch at #19301837 (0xb6c4b2e414fffbf406557ed0d8c6debfb80f0ef60cf3da34997e1a38155b6a09): got 0x951c3fb7179efa13b9a21e402f657e0ad21ef8d5f5045db7278621d071899f1c, expected 0xa991597d052a9df1e27b8bb5fde346a1e1aa93547b49347fd59b214bceb504a5

@krakovia-evm
Copy link
Author

hmmm...
after being caught again in the failed unwind loop, seems updating Lighthouse to latest version fixed the problem.
I'll check again in 24h but for now i'm getting the usual performance before that issue

@krakovia-evm
Copy link
Author

krakovia-evm commented Feb 26, 2024 •
edited
Loading

nvm, but i guess i found the problem.
seems the prune module is the RPC data
image

image

@krakovia-evm
Copy link
Author

back there.
I'll stop the docker and wait for a next upgrade or anything to try.
personally don't know what to try anymore ^^
image

@emhane
Copy link
Member

emhane commented Feb 26, 2024 •
edited
Loading

@emhane all those logs should be trace!

sure, maybe this one too 2024-02-23T13:21:52.756693Z DEBUG discv4: removed node node_id=0x.. @mattsse

@emhane
Copy link
Member

emhane commented Feb 27, 2024

@emhane all those logs should be trace!

sure, maybe this one too 2024-02-23T13:21:52.756693Z DEBUG discv4: removed node node_id=0x.. @mattsse

done #6815

@krakovia-evm krakovia-evm mentioned this issue Feb 28, 2024
Closed
@emhane emhane added A-staged-sync Related to staged sync (pipelines and stages) A-trie Related to Merkle Patricia Trie implementation labels Feb 29, 2024
@DaniPopes DaniPopes removed the S-needs-triage This issue needs to be labelled label Mar 6, 2024
@krakovia-evm
Copy link
Author

Problem has been fully resolved with the beta release!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-staged-sync Related to staged sync (pipelines and stages) A-trie Related to Merkle Patricia Trie implementation C-bug An unexpected or incorrect behavior
Projects
Reth Tracker
Archived in project
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mattsse @DaniPopes @emhane @krakovia-evm