Unequal number of senders

[nicolasochem]

Describe the bug

right after upgrade from alpha 10 to alpha 11, seeing thread panicks:

│ 2023-11-25T20:59:12.731886Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x52a1686229b3a4a0b475c5bbdb2bea4c48accb46f8513c90221e810e68086aa1 safe_block_hash=0x7d57f708f1e1 │
│ thread 'tokio-runtime-worker' panicked at crates/primitives/src/block.rs:58:9:                                                                                                             │
│ assertion `left == right` failed: Unequal number of senders                                                                                                                                │
│   left: 7                                                                                                                                                                                  │
│  right: 0 

However, I am succesfully synced and currently validating on holesky. So it seems to be spurious errors...

Steps to reproduce

Upgrade from 0.1.0-alpha 10 to 11.

Node logs

│ 2023-11-25T20:58:30.003383Z  INFO reth::cli: Status connected_peers=2 freelist=69891 latest_block=393753                                                                                   │
│ 2023-11-25T20:58:36.692505Z  INFO reth::node::events: Block added to canonical chain number=393754 hash=0x54c688da0fe278be2844dfd1e1ea11a2b350c677bf2d1005a76f6c71ea64baa6                 │
│ 2023-11-25T20:58:36.896961Z  INFO reth::node::events: Canonical chain committed number=393754 hash=0x54c688da0fe278be2844dfd1e1ea11a2b350c677bf2d1005a76f6c71ea64baa6 elapsed=3.196466ms   │
│ 2023-11-25T20:58:36.896983Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x54c688da0fe278be2844dfd1e1ea11a2b350c677bf2d1005a76f6c71ea64baa6 safe_block_hash=0x7d57f708f1e1 │
│ thread 'tokio-runtime-worker' panicked at crates/primitives/src/block.rs:58:9:                                                                                                             │
│ assertion `left == right` failed: Unequal number of senders                                                                                                                                │
│   left: 6                                                                                                                                                                                  │
│  right: 0                                                                                                                                                                                  │
│ 2023-11-25T20:58:50.434618Z  INFO reth::node::events: Block added to canonical chain number=393755 hash=0x7bf464f8b92b1172a655ff3fbd4fa2320ee3cac9dc2d842d9fbcfac1524e070a                 │
│ 2023-11-25T20:58:50.637697Z  INFO reth::node::events: Canonical chain committed number=393755 hash=0x7bf464f8b92b1172a655ff3fbd4fa2320ee3cac9dc2d842d9fbcfac1524e070a elapsed=3.455562ms   │
│ 2023-11-25T20:58:50.637721Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x7bf464f8b92b1172a655ff3fbd4fa2320ee3cac9dc2d842d9fbcfac1524e070a safe_block_hash=0x7d57f708f1e1 │
│ 2023-11-25T20:58:55.003121Z  INFO reth::cli: Status connected_peers=2 freelist=69891 latest_block=393755                                                                                   │
│ 2023-11-25T20:59:02.174069Z  INFO reth::node::events: Block added to canonical chain number=393756 hash=0xe2b20e007e946913086c5d3b2ae1b9c20ab377ce2033b0b7924a27dc7f20c190                 │
│ 2023-11-25T20:59:02.345070Z  INFO reth::node::events: Canonical chain committed number=393756 hash=0xe2b20e007e946913086c5d3b2ae1b9c20ab377ce2033b0b7924a27dc7f20c190 elapsed=3.430635ms   │
│ 2023-11-25T20:59:02.345101Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xe2b20e007e946913086c5d3b2ae1b9c20ab377ce2033b0b7924a27dc7f20c190 safe_block_hash=0x7d57f708f1e1 │
│ 2023-11-25T20:59:12.504506Z  INFO reth::node::events: Block added to canonical chain number=393757 hash=0x52a1686229b3a4a0b475c5bbdb2bea4c48accb46f8513c90221e810e68086aa1                 │
│ 2023-11-25T20:59:12.731861Z  INFO reth::node::events: Canonical chain committed number=393757 hash=0x52a1686229b3a4a0b475c5bbdb2bea4c48accb46f8513c90221e810e68086aa1 elapsed=4.790662ms   │
│ 2023-11-25T20:59:12.731886Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x52a1686229b3a4a0b475c5bbdb2bea4c48accb46f8513c90221e810e68086aa1 safe_block_hash=0x7d57f708f1e1 │
│ thread 'tokio-runtime-worker' panicked at crates/primitives/src/block.rs:58:9:                                                                                                             │
│ assertion `left == right` failed: Unequal number of senders                                                                                                                                │
│   left: 7                                                                                                                                                                                  │
│  right: 0                                                                                                                                                                                  │
│ 2023-11-25T20:59:20.003931Z  INFO reth::cli: Status connected_peers=2 freelist=69886 latest_block=393757                                                                                   │
│ 2023-11-25T20:59:25.326038Z  INFO reth::node::events: Block added to canonical chain number=393758 hash=0xaf51b10bef461f261d6fb31d130a4716240594a0a1dd5765b4f37f8c3696ee13                 │
│ 2023-11-25T20:59:25.554910Z  INFO reth::node::events: Canonical chain committed number=393758 hash=0xaf51b10bef461f261d6fb31d130a4716240594a0a1dd5765b4f37f8c3696ee13 elapsed=4.693189ms   │
│ 2023-11-25T20:59:25.554928Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xaf51b10bef461f261d6fb31d130a4716240594a0a1dd5765b4f37f8c3696ee13 safe_block_hash=0x7d57f708f1e1 │
│ 2023-11-25T20:59:25.555341Z  INFO reth::node::events: Pruner finished tip_block_number=393758 elapsed=260.036µs stats={SenderRecovery: (Finished, 0), Receipts: (Finished, 0), AccountHist │
│ 2023-11-25T20:59:37.732357Z  INFO reth::node::events: Block added to canonical chain number=393759 hash=0x68a969b5d68b159c4b5e3bc36758bb8bf54a0caee7d4a24e17e6261f8ac8359f                 │
│ 2023-11-25T20:59:37.955621Z  INFO reth::node::events: Canonical chain committed number=393759 hash=0x68a969b5d68b159c4b5e3bc36758bb8bf54a0caee7d4a24e17e6261f8ac8359f elapsed=2.150287ms   │
│ 2023-11-25T20:59:37.955643Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x68a969b5d68b159c4b5e3bc36758bb8bf54a0caee7d4a24e17e6261f8ac8359f safe_block_hash=0x7d57f708f1e1 │
│ thread 'tokio-runtime-worker' panicked at crates/primitives/src/block.rs:58:9:                                                                                                             │
│ assertion `left == right` failed: Unequal number of senders                                                                                                                                │
│   left: 4                                                                                                                                                                                  │
│  right: 0                                                                                                                                                                                  │
│ 2023-11-25T20:59:45.003472Z  INFO reth::cli: Status connected_peers=2 freelist=69886 latest_block=393759                                                                                   │
│ 2023-11-25T20:59:48.645263Z  INFO reth::node::events: Block added to canonical chain number=393760 hash=0xbcf98c8268b47df63d817661de9e8d86547c317740616fe1c9a4cf2af9998521                 │
│ 2023-11-25T20:59:48.820678Z  INFO reth::node::events: Canonical chain committed number=393760 hash=0xbcf98c8268b47df63d817661de9e8d86547c317740616fe1c9a4cf2af9998521 elapsed=4.985317ms   ││ 2023-11-25T20:59:48.820700Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xbcf98c8268b47df63d817661de9e8d86547c317740616fe1c9a4cf2af9998521 safe_block_hash=0x7d57f708f1e1 ││ 2023-11-25T21:00:00.675515Z  INFO reth::node::events: Block added to canonical chain number=393761 hash=0x4f10931fe9ab04c99c020af8b1b147f3c697efaf7a9e86bdccb379260faa430d                 │
│ 2023-11-25T21:00:00.860721Z  INFO reth::node::events: Canonical chain committed number=393761 hash=0x4f10931fe9ab04c99c020af8b1b147f3c697efaf7a9e86bdccb379260faa430d elapsed=4.411683ms   ││ 2023-11-25T21:00:00.860751Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x4f10931fe9ab04c99c020af8b1b147f3c697efaf7a9e86bdccb379260faa430d safe_block_hash=0x7d57f708f1e1 ││ 2023-11-25T21:00:10.003986Z  INFO reth::cli: Status connected_peers=2 freelist=69885 latest_block=393761                                                                                   │
│ 2023-11-25T21:00:12.314983Z  INFO reth::node::events: Block added to canonical chain number=393762 hash=0xafd44e563712f3f98e3a8666d3d46051b82d4664d5f66b7e9b89c8dce08801e4                 │
│ 2023-11-25T21:00:12.556759Z  INFO reth::node::events: Canonical chain committed number=393762 hash=0xafd44e563712f3f98e3a8666d3d46051b82d4664d5f66b7e9b89c8dce08801e4 elapsed=6.435553ms   │
│ 2023-11-25T21:00:12.556777Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xafd44e563712f3f98e3a8666d3d46051b82d4664d5f66b7e9b89c8dce08801e4 safe_block_hash=0x7d57f708f1e1 │
│ thread 'tokio-runtime-worker' panicked at crates/primitives/src/block.rs:58:9:                                                                                                             │
│ assertion `left == right` failed: Unequal number of senders                                                                                                                                │
│   left: 5                                                                                                                                                                                  │
│  right: 0                                                                                                                                                                                  │
│ 2023-11-25T21:00:26.110896Z  INFO reth::node::events: Block added to canonical chain number=393763 hash=0x250c86585914ae85214679023b4c7a87e87f920431977d5d7a7afd60a909b3bb                 │
│ 2023-11-25T21:00:26.283230Z  INFO reth::node::events: Canonical chain committed number=393763 hash=0x250c86585914ae85214679023b4c7a87e87f920431977d5d7a7afd60a909b3bb elapsed=2.836713ms   │
│ 2023-11-25T21:00:26.283254Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x250c86585914ae85214679023b4c7a87e87f920431977d5d7a7afd60a909b3bb safe_block_hash=0x7d57f708f1e1 │
│ 2023-11-25T21:00:26.283728Z  INFO reth::node::events: Pruner finished tip_block_number=393763 elapsed=345.938µs stats={SenderRecovery: (Finished, 0), Receipts: (Finished, 0), AccountHist │
│ 2023-11-25T21:00:35.003461Z  INFO reth::cli: Status connected_peers=2 freelist=69881 latest_block=393763                                                                                   │
│ 2023-11-25T21:00:37.751153Z  INFO reth::node::events: Block added to canonical chain number=393764 hash=0x91c246c25200dbff1520231257a2032b237573cb2f7658a735ff5db7c3efb9d3                 │
│ 2023-11-25T21:00:37.968670Z  INFO reth::node::events: Canonical chain committed number=393764 hash=0x91c246c25200dbff1520231257a2032b237573cb2f7658a735ff5db7c3efb9d3 elapsed=2.339863ms   │
│ 2023-11-25T21:00:37.968708Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x91c246c25200dbff1520231257a2032b237573cb2f7658a735ff5db7c3efb9d3 safe_block_hash=0x7d57f708f1e1 │
│ thread 'tokio-runtime-worker' panicked at crates/primitives/src/block.rs:58:9:                                                                                                             │
│ assertion `left == right` failed: Unequal number of senders                                                                                                                                │
│   left: 4                                                                                                                                                                                  │
│  right: 0            

Platform(s)

Linux (x86)

What version/commit are you on?

0.1.0-alpha11

What database version are you on?

I have no name!@reth-0:/$ reth db version --log.file.directory /tmp --datadir /data/  
Current database version: 1
Local database version: 1

What type of node are you running?

Pruned with custom reth.toml config

What prune config do you use, if any?

[stages.headers]
downloader_max_concurrent_requests = 100
downloader_min_concurrent_requests = 5
downloader_max_buffered_responses = 100
downloader_request_limit = 1000
commit_threshold = 10000

[stages.total_difficulty]
commit_threshold = 100000

[stages.bodies]
downloader_request_limit = 200
downloader_stream_batch_size = 1000
downloader_max_buffered_blocks_size_bytes = 2147483648
downloader_min_concurrent_requests = 5
downloader_max_concurrent_requests = 100

[stages.sender_recovery]
commit_threshold = 5000000

[stages.execution]
max_blocks = 500000
max_changes = 5000000
max_cumulative_gas = 1500000000000

[stages.account_hashing]
clean_threshold = 500000
commit_threshold = 100000

[stages.storage_hashing]
clean_threshold = 500000
commit_threshold = 100000

[stages.merkle]
clean_threshold = 50000

[stages.transaction_lookup]
commit_threshold = 5000000

[stages.index_account_history]
commit_threshold = 100000

[stages.index_storage_history]
commit_threshold = 100000

[peers]
refill_slots_interval = '5s'
trusted_nodes = []
connect_trusted_nodes_only = false
max_backoff_count = 5
ban_duration = '12h'

[peers.connection_info]
max_outbound = 100
max_inbound = 30
max_concurrent_outbound_dials = 10

[peers.reputation_weights]
bad_message = -16384
bad_block = -16384
bad_transactions = -16384
already_seen_transactions = 0
timeout = -4096
bad_protocol = -2147483648
failed_to_connect = -25600
dropped = -4096

[peers.backoff_durations]
low = '30s'
medium = '3m'
high = '15m'
max = '1h'

[sessions]
session_command_buffer = 32
session_event_buffer = 260

[sessions.limits]

[sessions.initial_internal_request_timeout]
secs = 20
nanos = 0

[sessions.protocol_breach_request_timeout]
secs = 120
nanos = 0

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct