TimeoutInfo replay will produce deadlock

[TheWinds]

Tendermint version (use tendermint version or git rev-parse --verify HEAD if installed from source):
v0.34.x
v0.35.x

ABCI app (name for built-in, URL for self-written if it's publicly available):
any

Environment:

• OS (e.g. from /etc/os-release):
• Install tools:
• Others:
  centos7

What happened:
The consensus module gets stuck when replaying the timeout message

What you expected to happen:

Have you tried the latest version: yes/no
yes

How to reproduce it (as minimally and precisely as possible):

Logs (paste a small part showing an error (< 10 lines) or link a pastebin, gist, etc. containing more of the log file):

Config (you can paste only the changes you've made):

node command runtime flags:

Please provide the output from the http://<ip>:<port>/dump_consensus_state RPC endpoint for consensus bugs
no，this api will get stuck

Anything else we need to know:

Maybe these codes should be promoted before catch up replay？

https://github.com/tendermint/tendermint/blob/master/internal/consensus/state.go#L433-L440

catchUpReplay method will call cs.scheduleTimeout,but timeoutRoutine not created,so

the following code will lock

func (t *timeoutTicker) ScheduleTimeout(ti timeoutInfo) {
	t.tickChan <- ti
}

[williambanfield]

Thanks for filing the issue @TheWinds, would you mind attaching some logs so from the process from when this happens?

[ancazamfir]

I was also running into this with node/ validator restart during consensus with multiple rounds. I can confirm that moving the cs.timeoutTicker.Start(ctx) before the replay loop solves the locking.

[TheWinds]

Thanks for filing the issue @TheWinds, would you mind attaching some logs so from the process from when this happens?

Because the replay process is blocked, no helpful logs are output. You can reproduce this problem with continuous writes at scale.

[TheWinds]

I was also running into this with node/ validator restart during consensus with multiple rounds. I can confirm that moving the cs.timeoutTicker.Start(ctx) before the replay loop solves the locking.

yes i did that too

[williambanfield]

backported to v0.35: #8082
backported to v0.34: #8079

[TheWinds]

good job