testnet rpc node state sync error

[zouyuxuan]

Describe the bug
the running test network nodes are in synchronization state for a long time
To Reproduce
when i run testnet with ./target/release/neard --home /data/neartest run ,when it started downloading headers to 99.5%, it began to be in the State 3steam1uecmpjejmxnjxsqxp8paupvpxk1pjk4xca3a, which lasted more than ten hours.
logs like this
1.5MiB/s 0.00 bps 0 gas/s CPU: 44%, Mem: 3.2 GiB
Dec 25 09:54:40.189 WARN sync: State sync didn't download the state for shard 1 in 60 seconds, sending StateRequest again
Dec 25 09:54:49.568 INFO stats: State 3StEAM1uEcMpjEJJMxNjXsqXP8PaUpVpXK1Pjk4xCa3A[0: header][1: header][2: header][3: header] 30/26/40 peers ⬇ 1.8MiB/s ⬆ 1.5MiB/s 0.00 bps 0 gas/s CPU: 53%, Mem: 3.2 GiB
Dec 25 09:54:59.573 INFO stats: State 3StEAM1uEcMpjEJJMxNjXsqXP8PaUpVpXK1Pjk4xCa3A[0: header][1: header][2: header][3: header] 30/26/40 peers ⬇ 954.2kiB/s ⬆ 661.0kiB/s 0.00 bps 0 gas/s CPU: 46%, Mem: 3.2 GiB
Dec 25 09:55:03.220 WARN sync: State sync didn't download the state for shard 0 in 60 seconds, sending StateRequest again
Dec 25 09:55:09.578 INFO stats: State 3StEAM1uEcMpjEJJMxNjXsqXP8PaUpVpXK1Pjk4xCa3A[0: header][1: header][2: header][3: header] 30/26/40 peers ⬇ 818.2kiB/s ⬆ 553.4kiB/s 0.00 bps 0 gas/s CPU: 34%, Mem: 3.2 GiB Dec 25 09:55:19.583 INFO stats: State 3StEAM1uEcMpjEJJMxNjXsqXP8PaUpVpXK1Pjk4xCa3A[0: header][1: header][2: header][3: header] 30/26/40 peers ⬇ 778.0kiB/s ⬆ 505.8kiB/s 0.00 bps 0 gas/s CPU: 27%, Mem: 3.2 GiB
Dec 25 09:55:29.588 INFO stats: State 3StEAM1uEcMpjEJJMxNjXsqXP8PaUpVpXK1Pjk4xCa3A[0: header][1: header][2: header][3: header] 30/26/40 peers ⬇ 743.6kiB/s ⬆ 482.2kiB/s 0.00 bps 0 gas/s CPU: 40%, Mem: 3.2 GiB
Dec 25 09:55:33.602 WARN sync: State sync didn't download the state for shard 3 in 60 seconds, sending StateRequest again Dec 25 09:55:34.050 WARN sync: State sync didn't download the state for shard 2 in 60 seconds, sending StateRequest again
Dec 25 09:55:39.592 INFO stats: State 3StEAM1uEcMpjEJJMxNjXsqXP8PaUpVpXK1Pjk4xCa3A[0: header][1: header][2: header][3: header] 30/26/40 peers ⬇ 741.5kiB/s ⬆ 479.6kiB/s 0.00 bps 0 gas/s CPU: 48%, Mem: 3.2 GiB
Dec 25 09:55:40.196 WARN sync: State sync didn't download the state for shard 1 in 60 seconds, sending StateRequest again
Dec 25 09:55:49.596 INFO stats: State 3StEAM1uEcMpjEJJMxNjXsqXP8PaUpVpXK1Pjk4xCa3A[0: header][1: header][2: header][3: header] 30/26/40 peers ⬇ 703.9kiB/s ⬆ 451.3kiB/s 0.00 bps 0 gas/s CPU: 41%, Mem: 3.2 GiB
Dec 25 09:55:59.601 INFO stats: State 3StEAM1uEcMpjEJJMxNjXsqXP8PaUpVpXK1Pjk4xCa3A[0: header][1: header][2: header][3: header] 30/26/40 peers ⬇ 689.3kiB/s ⬆ 440.9kiB/s 0.00 bps 0 gas/s CPU: 27%, Mem: 3.2 GiB
Dec 25 09:56:03.667 WARN sync: State sync didn't download the state for shard 0 in 60 seconds, sending StateRequest again
Dec 25 09:56:09.606 INFO stats: State 3StEAM1uEcMpjEJJMxNjXsqXP8PaUpVpXK1Pjk4xCa3A[0: header][1: header][2: header][3: header] 30/26/40 peers ⬇ 696.6kiB/s ⬆ 445.1kiB/s 0.00 bps 0 gas/s CPU: 52%, Mem: 3.2 GiB
Dec 25 09:56:19.610 INFO stats: State 3StEAM1uEcMpjEJJMxNjXsqXP8PaUpVpXK1Pjk4xCa3A[0: header][1: header][2: header][3: header] 30/26/40 peers ⬇ 689.9kiB/s ⬆ 443.2kiB/s 0.00 bps 0 gas/s CPU: 39%, Mem: 3.2 GiB
Dec 25 09:56:29.615 INFO stats: State 3StEAM1uEcMpjEJJMxNjXsqXP8PaUpVpXK1Pjk4xCa3A[0: header][1: header][2: header][3: header] 30/26/40 peers ⬇ 682.7kiB/s ⬆ 438.3kiB/s 0.00 bps 0 gas/s CPU: 38%, Mem: 3.2 GiB
Dec 25 09:56:33.605 WARN sync: State sync didn't download the state for shard 3 in 60 seconds, sending StateRequest again
Dec 25 09:56:34.055 WARN sync: State sync didn't download the state for shard 2 in 60 seconds, sending StateRequest again
Dec 25 09:56:39.621 INFO stats: State 3StEAM1uEcMpjEJJMxNjXsqXP8PaUpVpXK1Pjk4xCa3A[0: header][1: header][2: header][3: header] 30/26/40 peers ⬇ 683.1kiB/s ⬆ 439.1kiB/s 0.00 bps 0 gas/s CPU: 30%, Mem: 3.2 GiB
Dec 25 09:56:40.200 WARN sync: State sync didn't download the state for shard 1 in 60 seconds, sending StateRequest again
Dec 25 09:56:49.625 INFO stats: State 3StEAM1uEcMpjEJJMxNjXsqXP8PaUpVpXK1Pjk4xCa3A[0: header][1: header][2: header][3: header] 30/26/40 peers ⬇ 683.8kiB/s ⬆ 438.2kiB/s 0.00 bps 0 gas/s CPU: 47%, Mem: 3.2 GiB
Dec 25 09:56:59.630 INFO stats: State 3StEAM1uEcMpjEJJMxNjXsqXP8PaUpVpXK1Pjk4xCa3A[0: header][1: header][2: header][3: header] 30/26/40 peers ⬇ 704.8kiB/s ⬆ 454.5kiB/s 0.00 bps 0 gas/s CPU: 52%, Mem: 3.2 GiB

Expected behavior

Screenshots
If applicable, add screenshots to help explain your problem.

Version (please complete the following information):

• nearcore neard (release 1.23.0)
• rcargo version 1.56.0 (4ed5d137b 2021-10-04)
• testnet

Additional context
config.json like this
{
"genesis_file": "genesis.json",
"genesis_records_file": null,
"validator_key_file": "validator_key.json",
"node_key_file": "node_key.json",
"rpc": {
"addr": "0.0.0.0:3030",
"cors_allowed_origins": [
"*"
],
"polling_config": {
"polling_interval":

{ "secs": 0, "nanos": 500000000 }

,
"polling_timeout":

{ "secs": 10, "nanos": 0 }

}
},
"telemetry":

{ "endpoints": [ "https://explorer.testnet.near.org/api/nodes" ] }

,
"network": {
"addr": "0.0.0.0:24567",
"external_address": "",
"boot_nodes": "ed25519:4k9csx6zMiXy4waUvRMPTkEtAS2RFKLVScocR5HwN53P@34.73.25.182:24567,ed25519:4keFArc3M4SE1debUQWi3F1jiuFZSWThgVuA2Ja2p3Jv@34.94.158.10:24567,ed25519:D2t1KTLJuwKDhbcD9tMXcXaydMNykA99Cedz7SkJkdj2@35.234.138.23:24567,ed25519:CAzhtaUPrxCuwJoFzceebiThD9wBofzqqEMCiupZ4M3E@34.94.177.51:24567",
"max_peers": 40,
"handshake_timeout":

{ "secs": 20, "nanos": 0 }

,
"reconnect_delay":

{ "secs": 60, "nanos": 0 }

,
"skip_sync_wait": false,
"ban_window":

{ "secs": 10800, "nanos": 0 }

,
"blacklist": []
},
"consensus": {
"min_num_peers": 3,
"block_production_tracking_delay":

{ "secs": 0, "nanos": 100000000 }

,
"min_block_production_delay":

{ "secs": 0, "nanos": 600000000 }

,
"max_block_production_delay":

{ "secs": 2, "nanos": 0 }

,
"max_block_wait_delay":

{ "secs": 6, "nanos": 0 }

,
"reduce_wait_for_missing_block":

{ "secs": 0, "nanos": 100000000 }

,
"produce_empty_blocks": true,
"block_fetch_horizon": 50,
"state_fetch_horizon": 5,
"block_header_fetch_horizon": 50,
"catchup_step_period":

{ "secs": 0, "nanos": 100000000 }

,
"chunk_request_retry_period":

{ "secs": 0, "nanos": 200000000 }

,
"header_sync_initial_timeout":

{ "secs": 10, "nanos": 0 }

,
"header_sync_progress_timeout":

{ "secs": 2, "nanos": 0 }

,
"header_sync_stall_ban_timeout":

{ "secs": 120, "nanos": 0 }

,
"header_sync_expected_height_per_second": 10
},
"tracked_accounts": [],
"tracked_shards": [0],
"archive": false
}

[bowenwang1996]

This is normal. State sync may take several hours to finish and the node would not make any progress in the meantime.

[zouyuxuan]

It has been in this state for several days.after synchronization, the data size is only 132g, and its original data size is 153g

[stale]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months.
It will be closed in 7 days if no further activity occurs.
Thank you for your contributions.

[Longarithm]

Currently state sync doesn't work anyway, and we plan to improve it in #8545.