ABCI++ vote extensions & FinalizeBlock

[sergio-mena]

This is the continuation of the work started in tendermint/tendermint#9053: backport ABCI++ implementation, which lives in the v0.36.x branch.
Once this tracking issue is completed, the whole ABCI++ interface should be ready for QA process, and then release.

Proposed Path to Done. Summary

The main idea behind this plan is to proceed in the same way as tendermint/tendermint#9053.
The main approach is:

• We will "follow" our work on v0.36.x. By "following a PR", I mean looking at that PR (similar work done for v0.36.x) and copying over only what makes sense. "git cherry-pick" turned out to be our best friend for "following" PRs in ABCI++ Prepare/Process Proposal tendermint/tendermint#9053
• We will be considering only commits related to vote extensions and FinalizeBlock
• We will be skipping PrepareProposal/ProcessProposal related commits (as they are already in main via ABCI++ Prepare/Process Proposal tendermint/tendermint#9053)

The work is structured very similarly to #9053. It consists of two main parts (further described below):

• (1) Preliminary work
• (2) Core feature work

(1) and (2) can mostly proceed in parallel.

protobufs will be managed as in v0.37.x → They will evolve with the code

Proposed Path to Done. Details

(1) Preliminary Work

We could have two threads here, but as all these tasks serialized are shorter than the critical path in "Core Feature Work", there's no point

• Create feature branch: feature/abci++vef
• Restore CI on feature/abci++vef branch
• e2e tests: port any improvements not included in the core work section (git diff HEAD..v0.36.x -- test/e2e) (tracking: e2e: add contexts and e2e generator tests tendermint/tendermint#9426)

(2) Core feature work

We have three threads that can proceed in parallel:

• Vote extensions (*)
• FinalizeBlock (*)
• Spec and doc work

(*) Only when vote extensions and FinalizeBlock are done, can we proceed with the fourth part:

• Final adjustments
  • tasks here could be further parallelized

Vote Extensions

• Old vote extensions
  • Follow abci: Vote Extension 1 tendermint/tendermint#6646, then ABCI Vote Extension 2 tendermint/tendermint#6885
  • Double-check integration of PrepareProposal -- vote extensions
    • Check if there is something missing from abci: PrepareProposal-VoteExtension integration [2nd try] tendermint/tendermint#7821 (or PrepareProposal-VoteExtension integration tendermint/tendermint#6915)
• Sync vote extensions with Spec tendermint/tendermint#9835
  • First cherry-pick Vote extensions: new design tendermint/tendermint#8031 (ignore spec part),
  • Main part: follow abci++: Sync implementation and spec for vote extensions tendermint/tendermint#8141
• Vote extension cleanup tendermint/tendermint#9846
  • follow abci++: Vote extension cleanup tendermint/tendermint#8402
• Vote extension propagation Vote extension propagation tendermint/tendermint#9852
  • follow abci++: Propagate vote extensions (RFC 017) tendermint/tendermint#8433
• Vote extensions: Parameter activation logic tendermint/tendermint#9861
  • Add activation logic: follow abci++: add consensus parameter logic to control vote extension require height tendermint/tendermint#8547
• Vote extensions: Add parameter in proto tendermint/tendermint#9864
  • follow abci++: add proto fields for enabling vote extensions tendermint/tendermint#8587
• Integrate divergences between cherry-picked implementation and manual work in #9620 tendermint/tendermint#9887

FinalizeBlock

• abci: port across finalize block from v0.36 tendermint/tendermint#9427 [Estimation: 4 days]
  • Follow Completed the existing FinalizeBlock PR and rebased to master tendermint/tendermint#7798 (it's bulky, and might depend on other infra PRs, if yes cherry-pick them first)
• Sync FinalizeBlock with spec
  • Description: follow abci: Synchronize FinalizeBlock with the updated specification tendermint/tendermint#7983

Final adjustments

Most of these tasks can be done in parallel, but they are quite short.

• ABCI 2.0: Final adjustments #10

Spec and doc work

Low risk, can proceed completely in parallel with the rest

• ABCI 2.0: Spec work #24
• ABCI 2.0: Docs and tutorials #25

Original issue: tendermint/tendermint#9396

[adizere]

Closing this as it's pretty much done. The only item left is issue 25, which is tracked separately.