PROPOSAL: Lower block times in 2 phases

[Skarsom]

TL;DR-- We currently wait 5 seconds AFTER each block to let the validators who were not part of the 66% needed to create a block catch up. If we did not do this only 66% of stake would ever receive block rewards and the other 33% would be left behind. This would change that time from 5 seconds to 2 seconds.

NOTE: If the following proposal passes, nodes will be required to upgrade to a new binary by June 1st at 06:00 GMT. The new binary, while we could create a proposal that does not require it, is required under this proposal so as to provide a graceful error message to those who have not upgraded that they must. Otherwise, users who did not upgrade in time would simply stop performing their Consensus duties without an understanding of why their node broke.

This proposal fixes a bug that results in validators unnecessarily waiting idle for 5 seconds in between blocks due to timeout_commit being defaulted to 5 seconds. The timeout_commit parameter in node configurations measures how long nodes will wait only after a consensus has been reached before beginning the next block. This timeout ensures that validators other than the 2/3 who reached consensus have a chance to participate in stake rewards.

Fast block times are essential for the mainstream adoption of permissionless blockchains as true decentralized database replacements. Data collected from monitoring the ABCI state history via Mainnet gRPC of dozens of blocks indicates that 99/100 of the bonded validators will be able to keep up with this change without missing any blocks. We thus conclude that the probability of a user being on a validator that is removed from the consensus as a result from this is 1%, or negligible as compared to the benefits of significantly faster blocks.

This proposal would, as a side effect, result in increased stake rewards distribution per unit time. This is because the block time would be lowered by 3 seconds flat, and as such there would be more blocks per year.

A YES VOTE means that you are voting to decrease block times by ~3 seconds per block, and increasing the requirements for validators to maintain uptime, performance, and connectivity so as to not be punished for slowing down the network.

A NO VOTE means that you are voting to keep block times in their current state. Validators will not have stringent uptime and performance requirements.

Proposal execution is divided into two phases. This is to maintain compatibility with validators whose timeout_commit is still 5 seconds for a reasonable amount of time.

PHASE 1 (IMMEDIATELY UPON PASSING):

All validators MUST, as soon as possible upon this proposal passing, update the following parameters in their config.toml file:

• timeout_commit to 2s
• timeout_propose to 5s
• timeout_prevote to 3s

PHASE 2 (June 1st at 06:00:00 GMT):

(1) All validators MUST update the following parameters in their config.toml file:

• timeout_propose to 3s
• timeout_prevote to 1s
  (2) Crypto.com and the Crypto.org Foundation are directed to issue a release to chain-main with a commit by June 1st at 06:00:00 GMT that changes the default timeout_commit upon config.toml generation to 2s

NETWORK RISK:

The network will halt if and only if under two thirds of staked validators have completed Phase 1 by the Phase 2 deadline (June 1st at 06:00:00 GMT).

IBC compatibility is unaffected.

If over two thirds of staked validators have completed Phase 1 by the Phase 2 deadline, there is no risk of a network halt. As a reminder, the timeout_commit stage is only entered after a consensus is reached.

[Skarsom]

I spun up an AWS instance (because many nodes are on AWS so it uses Amazon's own backbone to reduce network latency) and ran a script that would query the ABCI Consensus state dumps of mainnet.crypto.org as well as multiple other public nodes that had it open, for 4 in total, in parallel.

These data were aggregated and resulted in the following conclusions over 5,000 measured Crypto.org Mainnet blocks:

• A mean of 99% (99/100) active validators were able to finish the round within 2 seconds (+/- 1)
• A mean of 65% (65/100) active validators were able to finish the round within 1 second (+/- 5)

Non-active validators were not measured by this script.

The following ABCI endpoint was used in all measurements: https://mainnet.crypto.org:26657/dump_consensus_state

[DarkCenobyte]

Could you share the list of validators addresses that wasn't able to finish the round within 2 seconds?

It could allow the validators that would be impact by this proposal to upgrade or improve their nodes before the proposal and reassure those who doubt their network/hardware.

(even if the proposal is 3 seconds, we can assume that in case of exceptional high CPU load or network problems, they would fall more easily than the majority of other validators)

[Skarsom]

@DarkCenobyte The information of which validator addresses did not finish was untracked by the script. However, you can access information on which validators have what state in the current consensus via the following gRPC endpoint: https://mainnet.crypto.org:26657/dump_consensus_state

Thanks for the question!

[morhaf-ko]

Hi @Skarsom, thank you for the proposal, could you please elaborate more on how can we use the link above to check the affected validators? what states I should pay attention to?

[Sdpierret]

The "blocks_per_year": "6311520", value must be changed by the proposal

[Skarsom]

Actually, this would lower the block time to an average of 4.5 seconds. This is already what blocks_per_year expects (it sees 5s but this would compensate for the time where it was 7.5 imo), and is why right now there is a lower APY than expected and we would be able to sustain 20% for over 10 years given we are not on pace to pay out 500M CRO this year in mainnet rewards.

So there is no need to change this parameter-- we are essentially compensating for it by instead of lowering the parameter, we are lowering block times.

[tomtau]

Thank you for the proposal -- the default config was already partially adjusted: https://github.com/crypto-org-chain/chain-main/pull/494/files

If you also intend to submit this proposal for on-chain voting, it'll be good to clarify a few items for the full context:

1. How is this done on other networks with Tendermint Core? The config default values come out of Tendermint Core, so any Cosmos SDK-based network will potentially share them and it'll be good to know if this was tackled or discussed elsewhere (and how?).

2. What is the reasoning behind the default values? Not sure if they are arbitrary / results of some development test timeouts or there was some more detailed analysis.

3. What is the behaviour under more adversarial network conditions? Congestions, power blackouts, network partitioning etc.

4. What is the impact on IBC deployments? Note that IBC business logic operates with timeoutHeight and timeoutTimestamp https://github.com/cosmos/ibc/tree/master/spec/core/ics-004-channel-and-packet-semantics -- e.g. for the Tendermint (light) client implementation, I don't know if the Rust/TS relayer code operates with assuming these default values in some way.

[Skarsom]

To summarize the answers I put down below:

• Thank you for the questions. Important and reflect questions the community has about this moving forward.

1. ATOM does 5s, Binance Coin does 1s. But I would rather not focus on that since I believe the only way to move forward is to look forward and not sideways.
2. Default values reasoning is to allow as many validators as possible reasonable time to submit a commit to get block rewards even after consensus is reached. My proposal is of the philosophy that we should be maximizing network scalability and user experience, at the detriment of lax validator requirements.
3. No effect whatsoever. timeout_commit only affects the phase after consensus is reached, and adversarial network conditions would affect the reaching of a consensus in the first place (this would be timeout_precommit).
4. No effect whatsoever. These two parameters are per-transaction and optional, and as such users who choose to specify them would simply do them in accordance with the average block time on our chain rather than randomly choosing something like current_height+1.

[tomtau]

for 4., "No effect whatsoever" is in theory -- or have you looked into the implementation? What I meant whether to also consider that relayer code may need a patch for that (as in the past, it hardcoded things like coin type or assumed chain-id only having one dash... so there may be an effect)

[Skarsom]

@tomtau This is a great observation about cosmos, actually! You are correct in that there is a hardcoded value in the relayer code that the IBC developers themselves failed to point out to me upon being asked. https://github.com/cosmos/relayer/blob/fd7e7b5abec12d23ca29d0be47b9dad478d5e821/relayer/packet-tx.go#L35

As you can see here, if the initiating user of the IBC transaction does not specify a timeout time nor a timeout block height, the default will be 1,000 blocks into the future.

Our current block time is 6.4s according to Cros Nest data, and this would decrease it to 3.4s. So this leaves 3,400 seconds or 56 minutes as the default timeout. More than enough for an IBC transfer, so both in theory and in practice this will be fine.

Thanks again for the comment and illustrating this point about Cosmos hardcoding nonsensical defaults sometimes.

[Skarsom]

I will also loop in the IBC team at Cosmos again to make sure this is documented for future reference.

[JayT106]

To shorten the block commit timeout, It might need to consider the impact to the db layer during the block execution (especially the CosmWasm transaction allow in the feature?) and the total DB scales. When the DB states become a huge amount (we might be able to prune it) the changes might lead to the commit failed and the consensus will be halt (or break).

[tomtau]

That's a good point too -- doing the experiment @Skarsom posted #522 (comment) now (i.e. roughly one month of the network being alive) vs doing it when there are more features (e.g. cosmwasm https://github.com/crypto-org-chain/chain-main/blob/master/doc/architecture/adr-001.md ) and/or the internal storage is larger.

[Skarsom]

A larger DB layer would only affect the precommit stage. This is out-of-scope of the current proposal and discussion, as this affects the stage after a consensus (2/3 of stake in validators) have precommitted.

Let me use an example:

Say each commit now takes 5 seconds to run because of the huge CosmWasm transactions and unpacking requirements per transaction (this would be absolutely insane and would never happen, btw).

The timeout_precommit stage will now take +5s because NO validators will have submitted a precommit before 5 seconds is up, as everything is slower. So timeout_commit won't even be entered until 5 seconds later than it normally would be. Does this make sense?

[Skarsom]

@tomtau Thank you for your questions. There is a lot of misunderstanding at what timeout_commit is and how it works, and I am happy to clarify both that and your questions. Let me address these, and I will also address them in editing the original post and in the proposal that will go on-chain:

1. I must note that timeout_commit is only entered after 2/3 of all validators have submitted their precommits.

"timeout-commit = how long we wait after committing a block, before starting on the new height (this gives us a chance to receive some more precommits, even though we already have +2/3)" (emphasis added)

2. ATOM uses 5s. Binance Chain/BSC uses 1s and also has skip_timeout_commit = true (source: https://github.com/binance-chain/node-binary/blob/master/fullnode/prod/0.6.0/config/config.toml#L261).

3. These default values are meant to reflect the broader goal of Tendermint: to enable developers to develop consensus-based applications that change the world. Importantly, their goal is different from Crypto.org-- we are attempting to reinvent the world's financial system and put cryptocurrency in every wallet, NOT solely to provide tools for enthusiasts and developers. Essentially, there was no need for <1s block times on all Tendermint networks, because they were not obtaining a Visa partnership or attempting to put the whole world on ATOM.

4. Like I said in Problem: no contribution guidelines and templates #1, there is NO impact under adversarial network conditions to the state of the network. The worst possible impact is 1/3 of all validators being unable to precommit. That is literally the worst case scenario, as timeout_commit is only entered after consensus is reached. I chose 2s in this proposal because I thought it would meet the best of both worlds: generous enough to all 99% of all validators to continue to maintain their stake rewards and role in the network, but strict enough to move the network forward.

5. There is no impact on IBC, as IBC does not require a specific block time or times.
   "The timeoutHeight indicates a consensus height on the destination chain after which the packet will no longer be processed, and will instead count as having timed-out.
   The timeoutTimestamp indicates a timestamp on the destination chain after which the packet will no longer be processed, and will instead count as having timed-out." (source: https://github.com/cosmos/ibc/blob/0eba039ed65a66eace21124041d39be07ebfb69a/spec/core/ics-004-channel-and-packet-semantics/README.md)

As you can see, these parameters specify a height and/or timestamp on the destination chain. These are the only timeout-related parameters, and there is no default assumed in the Rust code.

6. JayT106's point regarding the database layer is a point about Cosmos-SDK itself, and is unrelated to timeout_commit and as such this proposal. As I said, timeout_commit only runs after a consensus is already reached. His question regarding the state storage layer involves how long it takes to reach a consensus in the first place.

[Skarsom]

To further address the IBC case, since this is extremely critical for Crypto.org:

"When non-zero, these timeout values determine the deadline before which the receiving module must process a packet." (https://github.com/cosmos/cosmos-sdk/blob/9fd866e3820b3510010ae172b682d71594cd8c14/docs/ibc/overview.md)

The user can choose to specify these when creating an IBC transaction. There are no defaults or automatic values for these parameters entered.

[tomtau]

Is there any reason why the proposal was submitted on-chain as SoftwareUpgradeProposal instead of TextProposal and the proposal's plan height is set as "0" and "info" doesn't contain the necessary information for software upgrade (e.g. for auto-download on full nodes)? This doesn't seem to match the intended execution in the proposal description and will have an unfortunate side-effect: SoftwareUpgradeProposal is intended for hardfork-like situations (i.e. new or breaking functionality) which this proposal doesn't seem to require.

[Skarsom]

1. It is a SoftwareUpgradeProposal because we intend for CDC to create a new release with an upgrade handler that makes no state store changes and simply returns upgraded = true. This is because a failure to find an upgrade handler at a specified time provides a graceful panic, rather than the mess of rounds that would ensue if a node or validator had not changed their timeout_commit from 5s because they did not know the proposal existed. The result would be an almost certain support inquiry via Discord on why they were suddenly voting NIL constantly because they are on a completely different step of the round.

2. We are currently on Cosmos SDK 0.42.x, not 0.43.x, meaning upgrade plans support EITHER a height OR a time. The time is set to 2021-06-01 06:00:00 GMT. Height is 0 because we are not using a set block height. This is simply how proposals work on Cosmos chains. Refer to https://crypto.org/explorer/croeseid/tx/14743A3D7C281044DC5A9F73D51423FE2F492AC789BBCC564EE85DC53E7A353D.

3. info contains everything necessary. Upgrade handlers are registered in the new software. Not in the plan.

4. See Problem: no contribution guidelines and templates #1 regarding breaking validators who have not upgraded in time. This is not an unfortunate side effect. In fact, there is no side effect at all, other than the fact that validators who have not upgraded in time will receive a graceful error message requiring an upgrade rather than a slew of timeout errors.

[Skarsom]

To illustrate further.

Assume node A has timeout_commit of 5s, but the rest of the network has timeout_commit of 2s.

Node A will still be in timeout_commit by the time the rest of the network is in precommit, meaning they will miss the next round. Then they will be early to the follow-up round. If they are the proposer it will result in a new round being initiated, causing 2 rounds for that block. Or they will basically just participate in 2/5 of rounds.

It is a non-graceful and bad user experience. I am of the belief that we can make this upgrade painless, even for those who are not paying attention to upgrades, by providing graceful panic messages through an upgrade handler that does not perform hardfork-like state changes. I understand what the module was intended for, and I believe it is suitable here for these reasons. If users disagree they will vote no as is the purpose of governance.

We are making a chain that is intending to bring cryptocurrency to every wallet-- not just to enthusiasts' wallets. This means ensuring a world-class user experience for all of our users.

[tomtau]

It is a SoftwareUpgradeProposal because we intend for CDC to create a new release with an upgrade handler that makes no state store changes and simply returns upgraded = true

This is not mentioned in the original proposal description here -- it doesn't contain a commit revision when this should happen as well as the name of the intended upgrade handler, so it is ambiguous what is being voted for (the two phase execution with validators changing their config rather implies TextProposal and the current proposal looks malformed). Instead of retrofitting software upgrade post-hoc (deducing it from the upgrade plan), it'd be preferable in this order:

1. the proposal is discussed here and all context is cleared, and the execution is planned accordingly;
2. changes are applied to the code according to the proposal and a new binary is released;
3. the new binary upgrade is tested on the public testnet as well as in mainnet-like dry deployments;
4. the proposal is submitted on-chain.

Given one other software upgrade proposal with actual breaking changes is planned around this time, the timing and the proposal type of the proposal id 3 are unfortunate.

This is not an unfortunate side effect. In fact, there is no side effect at all

The side-effect is a need for a new binary despite no (needed) breaking changes.
Given all of the context, it'd be preferable if the proposal was submitted as TextProposal and if it passes, the validators are expected to apply Phase 2 config changes with the next actual software upgrade.

[Skarsom]

it doesn't contain a commit revision when this should happen as well as the name of the intended upgrade handler

This does not make sense to me. The name of an upgrade handler is the title of a software upgrade plan. Commits are not intended to be included in this section. The name is "Reduce timeout_commit in 2 phases for faster blocks". https://crypto.org/explorer/tx/AA75C0ED751DDBC99E33EB1CA23DC2576AA97F493C74AB39277671B87D024683

so it is ambiguous what is being voted for (the two phase execution with validators changing their config rather implies TextProposal and the current proposal looks malformed).

The fact that it is a SoftwareUpgradeProposal makes it unambiguous that it is not a TextProposal. I will nonetheless edit the original description accordingly. Thank you for the feedback in improving the chain's transparency. It is not malformed. I believe you may be using software that inaccurately displays the proposal -- namely, the Chain Desktop Wallet and Explorer do not correctly display all available plan information.

it'd be preferable in this order:

If Crypto.com would like a specific procedure to generally be followed for governance, it should outline it. As a user, I welcome this type of feedback, and ask that it be made clearly and transparently for future reference-- not after governance has launched.

I appreciate the feedback, and look forward to seeing whether users of the Crypto.org Chain decide whether to move forward with this unique type of SoftwareUpgradeProposal.

Given one other software upgrade proposal with actual breaking changes is planned around this time, the timing and the proposal type of the proposal id 3 are unfortunate.

I strongly disagree with your characterization that this proposal creates some conflict amongst the Crypto.org Foundation's ability to perform their duties, or would interfere with normal chain operations. Rather, it fixes a bug accidentally introduced by a Crypto.com engineer and ad-hoc semi-fixed a couple weeks ago by @lezzokafka in a manner that, I would note, leaves open ABCI sync errors across validators given the different timeouts across nodes.

A new binary would have a single line so as to denote a graceful panic shutdown to non-upgraded clients:
app.upgradeKeeper.SetUpgradeHandler("Reduce timeout_commit in 2 phases for faster blocks", func(ctx sdk.Context, plan upgrade.Plan) {})

Furthermore, this is exactly the type of Software Upgrade that occurred during Crossfire as proposed by Crypto.com: c1c8483#diff-0f1d2976054440336a576d47a44a37b80cdf6701dd9113012bce0e3c425819b7R433

it'd be preferable if the proposal was submitted as TextProposal and if it passes, the validators are expected to apply Phase 2 config changes with the next actual software upgrade.

The result would be that users who fail to pay attention receive ABCI errors and are unable to complete rounds without verbose error messages versus receiving a graceful panic that they need to update their node. In my personal opinion, a graceful panic via a null upgrade handler is better. I disagree with your characterization that "it'd be preferable", because that is a matter of opinion in which we differ, and is part of the reason why decentralized governance exists in the first place. It is up to users to determine whether or not they agree.

[Skarsom]

@tomtau I have clearly noted the behavior of this proposal as it relates to software upgrades in large text at the top of this GitHub discussion. I have also copied and pasted the note onto the Crypto.org Chain Discord. Users are welcome to change their vote if they feel misled. I will remain fully transparent on this proposal-- indeed, it is on-chain and thus cannot be untransparent-- and will remain to answer questions as they come up.

Users will vote as they see fit given the proposal as-is. If they think requiring a new binary is an improper mechanism to provide a better UX to outdated nodes, they may vote no.

[tomtau]

OK, thanks for clarifying the proposal. As for "a specific procedure to generally be followed for governance" mentioned in #522 (reply in thread) -- yes, it's currently tracked on this open issue: crypto-org-chain/chain-docs#158 you can possibly post some thoughts on that issue.

People may still ignore those, but the intention is to have governance procedure guidelines that would help to solidify each proposal before it's submitted on the network. We currently track the planned breaking changes / bigger architectural changes in https://github.com/crypto-org-chain/chain-main/tree/master/doc/architecture that may end up in SoftwareUpgradeProposal (e.g. https://github.com/crypto-org-chain/chain-main/blob/master/doc/architecture/adr-003.md )

For the timing, consolidating breaking changes, TextProposal or ParamChangeProposal,
it may be useful to have a bit different venue (perhaps similar to https://github.com/cosmos/cips/blob/master/CIPS/cip-1.md or something simpler in the beginning).

[Skarsom]

Thank you for informing me of the working issue for governance procedures. This is important as we both agree, and I will do my best to put some thoughts there in the coming days.

I think it is important that we set up a coherent system for proposals so that we can make governance as transparent and clean for users as possible. I intend to follow them all, but am more raising the point that users cannot follow non-existent procedures.

I think the CIP idea for adding a venue illustrating consolidating breaking changes makes sense. I also think this touches on something a bit different, which is TextProposals that outline a breaking change but not a breaking binary change. In this case, a simple warning message that is outlined within the TextProposal could show after a certain time if specified, but not a panic. Definitely some more work to be done in this regard in my opinion.

Thanks for the fruitful discussion!

[eeldivad]

i don't think they will notice a 1 second different whether we cut it down by 3 or 2 seconds from the user perspective. 9 years instead of 10 is less of an impact and wouldn't make much of a difference if we don't end up updating blocks_per_year. i'm leaning towards yes, but maybe it's better to do this later when we have more tx per block. i feel like we're burning through blocks so fast without much tx that we're increase sizes of full nodes in the long run unnecessarily. just something to consider although i don't think its a big factor

[lezzokafka]

On our journey of building a fully decentralized, open-source public chain --
Valuable suggestions, involvement and strong commitment from the community have been well received and appreciated by the team since we launched Crypto.org mainnet on 25th of March, 2021.

Since every governance proposal could impact different stakeholders of the Crypto.org chain, especially the validators and users.
After extensive evaluation, we found the impact of this proposal is not yet known and the suggested changes could potentially affect the regular operation of the current Crypto.org mainnet.

Thus, for the stability and availability of the Crypto.org chain mainnet :
We believe that it would be best to consider getting it tested on the Croeseid Testnet before applying the proposed changes to the mainnet.

Timeline wise, we would like to suggest:

• Firstly, apply the proposed parameters changes to the validators on the upcoming Crossed testnet upgrade (v2.0.0-croeseid). Then we will be able to evaluate the impact of the suggested parameters changes on the average block time and also other aspects such as stability of the IBC relayer;
• Afterwards, proposed parameters could be included in the upcoming mainnet upgrade as a recommendation, given that promising results were provided during the test.

[Skarsom]

@lezzokafka can Crypto.com please outline how it believes this could affect regular operation of the current Crypto.org Chain Mainnet?

We have a week before voting ends and CDC already had a chance to look at this proposal, including the fact it was a SoftwareUpgradeProposal, as the proposal itself was proposed on the Testnet weeks ago.

How about Crypto.com updates the Testnet now to include the proposal's changes? There is a week that Crypto.com has to vote no if the results are unsatisfactory, and CancelSoftwareUpgrade proposals also exist.

We have the time to test this, and I agree-- let's do it.

[Skarsom]

cc @tomtau on the above point regarding implementing it on the Testnet now so we can test the proposal. Let's do this. We have the testnet, it's a simple configuration change.

[Galadrin]

Hello, As a Cros-nest operator on testnet, I applied the PHASE 1 parameters on our node.

After 24 hours of operation, it results in 0 missed blocks.

chain-maind q slashing signing-info tcrocnclconspub1zcjduepqcwwerel8f7dfs34hkaqgvdspnjfznvjk87cjwve3mdjfka0snw2sgpcdsm --node="https://testnet-croeseid.crypto.org:26657"
address: tcrocnclcons174xs3uzalje8yplrvphuekndljc34d4tgcdvj0
index_offset: "174343"
jailed_until: "2021-04-30T09:54:00.922184465Z"
missed_blocks_counter: "0"
start_height: "217113"
tombstoned: false

[kwunyeung]

The signing performance will be affected by the traffic of the network as well. To see if validators can broadcast votes on time, this should be simulated with stress test.

[kwunyeung]

This proposal fixes a bug that results in validators unnecessarily waiting idle for 5 seconds in between blocks due to timeout_commit being defaulted to 5 seconds.

I'm confused with the wordings. This is a behaviour in Tendermint Core which is not a bug. The timeout_commit is for a node to wait for more signatures before commit. I also wonder why an upgrade is needed if the proposal has passed. This should be a setting change of validator nodes if we come to a consensus on block time. The default change in generating config.toml should not be state or consensus breaking. The chain should not halt even if the validators not updating the timeout_commit.