stabilization-equivalent-proofs #6686

AdoAdoAdo · 2024-12-19T13:57:11Z

Reasoning behind the pull request

Bugfixes and changes while stabilizing the new consensus (v2) based on equivalent proofs

Proposed changes

change from epoch start trigger to generic epoch notifier to trigger the transition to new consensus - simpler and covers more cases
in case of generic epoch notifier the validators need to also need to trigger the transition correctly
early exit from consensus v1 operation if a transition to v2 happens.
adapt fork detector probable highest nonce computation
fix intercepted equivalent proof checks
adapt and fix tests

Testing procedure

Pre-requisites

Based on the Contributing Guidelines the PR author and the reviewers must check the following requirements are met:

was the PR targeted to the correct branch?
if this is a larger feature that probably needs more than one PR, is there a feat branch created?
if this is a feat branch merging, do all satellite projects have a proper tag inside go.mod?

…new consensus version

… tests

…tabilization

…ock-sync Check proof on activation block sync

…tiversx/mx-chain-go into equivalent-proofs-feat-stabilization

sstanculeanu · 2025-01-10T16:32:34Z

consensus/spos/bls/proxy/subroundsHandler.go

@@ -57,6 +56,13 @@ type SubroundsHandler struct {
 	currentConsensusType consensusStateMachineType
 }

+func (s *SubroundsHandler) EpochConfirmed(epoch uint32, _ uint64) {


missing comment

sstanculeanu · 2025-01-10T16:44:46Z

...dOfEpoch/epochChangeWithNodesShufflingAndRater/epochChangeWithNodesShufflingAndRater_test.go

@@ -17,6 +17,7 @@ func TestEpochChangeWithNodesShufflingAndRater(t *testing.T) {
 		t.Skip("this is not a short test")
 	}

+	_ = logger.SetLogLevel("*:DEBUG")


not to forget removing this

sstanculeanu · 2025-01-10T16:45:24Z

integrationTests/sync/basicSync/basicSync_test.go

@@ -203,12 +203,16 @@ func TestSyncWorksInShard_EmptyBlocksNoForks_With_EquivalentProofs(t *testing.T)
 		t.Skip("this is not a short test")
 	}

+	_ = logger.SetLogLevel("*:DEBUG,process:TRACE,consensus:TRACE")


not to forget removing this

sstanculeanu · 2025-01-10T16:57:11Z

process/interceptors/interceptedDataVerifier.go

@@ -56,6 +56,7 @@ func (idv *interceptedDataVerifier) Verify(interceptedData process.InterceptedDa

 	err := interceptedData.CheckValidity()
 	if err != nil {
+		log.Debug("Intercepted data is invalid", "hash", interceptedData.Hash(), "err", err)


I think we'll need a special handle here, as interceptedData.Hash() will be the header hash, thus receiving an invalid proof for the first time will lead to always returning ErrInvalidInterceptedData for further proofs(even valid ones)

good find, will fix

sstanculeanu · 2025-01-10T16:57:53Z

process/interface.go

@@ -384,6 +384,7 @@ type ForkDetector interface {
 	GetNotarizedHeaderHash(nonce uint64) []byte
 	ResetProbableHighestNonce()
 	SetFinalToLastCheckpoint()
+	// ReceivedProof()


not to forget removing this

sstanculeanu · 2025-01-10T17:11:05Z

process/sync/baseSync.go

@@ -161,6 +162,10 @@ func (boot *baseBootstrap) requestedHeaderHash() []byte {
 	return boot.headerhash
 }

+func (boot *baseBootstrap) processReceivedProof(headerProof data.HeaderProofHandler) {


not to forget the implementation

sstanculeanu · 2025-01-10T17:11:58Z

scripts/testnet/variables.sh

@@ -52,13 +52,13 @@ export GENESIS_STAKE_TYPE="direct" #'delegated' or 'direct' as in direct stake
 export OBSERVERS_ANTIFLOOD_DISABLE=0

 # Shard structure
-export SHARDCOUNT=2
-export SHARD_VALIDATORCOUNT=3
+export SHARDCOUNT=1


this should be reverted

added checks on proofs parameters like epoch, nonce and shard

ssd04 · 2025-01-13T12:12:19Z

consensus/spos/bls/proxy/subroundsHandler.go

+func (s *SubroundsHandler) EpochConfirmed(epoch uint32, _ uint64) {
+	err := s.initSubroundsForEpoch(epoch)
+	if err != nil {
+		log.Error("SubroundsHandler.EpochStartAction: cannot initialize subrounds", "error", err)


Suggested change

log.Error("SubroundsHandler.EpochStartAction: cannot initialize subrounds", "error", err)

log.Error("SubroundsHandler.EpochConfirmed: cannot initialize subrounds", "error", err)

ssd04 · 2025-01-14T06:29:48Z

integrationTests/testProcessorNode.go

+	if tpn.EnableEpochsHandler.IsFlagEnabledInEpoch(common.EquivalentMessagesFlag, blockHeader.GetEpoch()) {
+
+		// first block after genesis does not have the previous proof, as well as first block after epoch change where the flag gets activated
+		shouldAddProof := blockHeader.GetNonce() > 1 && !common.IsEpochChangeBlockForFlagActivation(blockHeader, tpn.EnableEpochsHandler, common.EquivalentMessagesFlag)


use here common.ShouldBlockHavePrevProof?

…tion' into equivalent-proofs-feat-stabilization

sstanculeanu · 2025-01-20T09:47:34Z

epochStart/shardchain/trigger.go

+	if !t.newEpochHdrReceived && !metaHdr.IsStartOfEpochBlock() {
+		return
+	}


not needed, already part of shouldUpdateTrigger

sstanculeanu · 2025-01-20T09:49:52Z

process/block/baseProcess.go

@@ -625,7 +635,7 @@ func (bp *baseProcessor) sortHeadersForCurrentBlockByNonce(usedInBlock bool) (ma
 		}

 		if bp.hasMissingProof(headerInfo, hdrHash) {
-			return nil, fmt.Errorf("%w for header with hash %s", process.ErrMissingHeaderProof, hdrHash)
+			return nil, fmt.Errorf("%w for header with hash %s", process.ErrMissingHeaderProof, hex.EncodeToString([]byte(hdrHash)))


same fix is needed on L663 too

…es, fixes after review

ssd04 · 2025-01-20T10:05:46Z

process/block/baseProcess.go

@@ -2200,3 +2201,28 @@ func (bp *baseProcessor) checkSentSignaturesAtCommitTime(header data.HeaderHandl

 	return nil
 }
+
+func isProofEmpty(proof data.HeaderProofHandler) bool {


we can add this check when adding a new proof, we should not add an empty proof to pool

I think the check is already being done on the interceptor in the integrity check
func (iep *interceptedEquivalentProof) integrity() error

removed the check here

ssd04 · 2025-01-20T10:15:55Z

epochStart/shardchain/trigger.go

@@ -270,7 +276,9 @@ func NewEpochStartTrigger(args *ArgsShardEpochStartTrigger) (*trigger, error) {
 		extraDelayForRequestBlockInfo: args.ExtraDelayForRequestBlockInfo,
 	}

+	// TODO: for equivalent messages this trigger needs to be changed, on start of epoch block proof instead


is this comment still needed?

no, removed

ssd04 · 2025-01-20T10:18:01Z

consensus/spos/bls/v2/subroundBlock.go

 		return
 	}

 	headerLeader, err := sr.getLeaderForHeader(headerHandler)
 	if err != nil {
+		log.Debug("subroundBlock.receivedBlockHeader - error", err.Error())


error getting leader for header?

AdoAdoAdo added 2 commits December 19, 2024 15:55

use EpochNotifier instead of EpochStartNotifier to transition to the …

be02d30

…new consensus version

trigger consensus version change as validator

97b9e06

AdoAdoAdo changed the title ~~use EpochNotifier instead of EpochStartNotifier to transition to the …~~ stabilization-equivalent-proofs Dec 20, 2024

AdoAdoAdo added 4 commits December 20, 2024 10:14

temp change of equivalent messages enable epoch to shorten transition…

ccdd6cb

… tests

fixes integration tests

75e042a

Merge branch 'feat/equivalent-messages' into equivalent-proofs-feat-s…

6092d21

…tabilization

fixes integration tests

5cd3d46

AdoAdoAdo self-assigned this Dec 20, 2024

AdoAdoAdo and others added 21 commits December 23, 2024 14:49

fixes check proof on first block after genesis

330eb4a

fix header finality check for equivalent proofs

7832305

adapt integration tests

60584a0

adapt integration tests

a8fb6f1

fixes & debug logs

df47f28

fix eqv proofs epoch cfg for chain simulator tests

492631c

fix interceptor save unit test

84c0b98

fixes header prev proof verification

700f252

avoid deadlock when switching consensus

b03af58

remove check on first block after activation

b1be55c

integrate header proof round

f695092

add test for equivalent proof on shard block sync

13d6458

Merge pull request #6697 from multiversx/check-proof-on-activation-bl…

bbbeb98

…ock-sync Check proof on activation block sync

adapt fork detector and highest probable nonce for equivalent proofs

5f650fb

Merge branch 'equivalent-proofs-feat-stabilization' of github.com:mul…

b7d1990

…tiversx/mx-chain-go into equivalent-proofs-feat-stabilization

add the round to the header info

2211c74

update tests

f1a55a9

update more tests

958bf29

added checks on proofs parameters like epoch, nonce and shard

1235b89

fixes after review

d4243bc

small refactor + extended checks

681f506

sstanculeanu reviewed Jan 10, 2025

View reviewed changes

sstanculeanu and others added 2 commits January 13, 2025 18:36

Merge pull request #6698 from multiversx/check_proof_validity

5495467

added checks on proofs parameters like epoch, nonce and shard

fix base fork detector process received block

9c0a622

ssd04 reviewed Jan 14, 2025

View reviewed changes

AdoAdoAdo added 4 commits January 14, 2025 11:34

fix fork detector with equivalent proofs

be77f2d

Merge remote-tracking branch 'origin/equivalent-proofs-feat-stabiliza…

c3333bb

…tion' into equivalent-proofs-feat-stabilization

fix intercepted proof checks

828708c

fixes after review

747cd55

AdoAdoAdo marked this pull request as ready for review January 14, 2025 13:25

fix check for accepting a proof on interceptor

6e25ca3

sstanculeanu previously approved these changes Jan 14, 2025

View reviewed changes

ssd04 previously approved these changes Jan 14, 2025

View reviewed changes

remove v1 consensus callbacks from worker after v2 activation

25276b1

AdoAdoAdo dismissed stale reviews from ssd04 and sstanculeanu via 25276b1 January 15, 2025 10:51

AdoAdoAdo added 8 commits January 15, 2025 15:53

consensus v2 fixes

f61ee57

move setting of prev proof in processor and fix rating update

6a11dbe

fix checkProofsForShardData and shard transition trigger

0abb837

update header validator to cross check proof

aef1bc8

update shard epoch strart trigger for consensus proofs

281141b

fix linter issues

3126444

fix tests

30ba777

fix trigger registry test

a8378a3

sstanculeanu reviewed Jan 20, 2025

View reviewed changes

remove unused storer in intercepted proofs, revert variables.sh chang…

e63d4fe

…es, fixes after review

ssd04 reviewed Jan 20, 2025

View reviewed changes

fixes after review

3bd8598

sstanculeanu approved these changes Jan 20, 2025

View reviewed changes

ssd04 approved these changes Jan 20, 2025

View reviewed changes

AdoAdoAdo merged commit fd92b5a into feat/equivalent-messages Jan 20, 2025
5 checks passed

AdoAdoAdo deleted the equivalent-proofs-feat-stabilization branch January 20, 2025 11:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stabilization-equivalent-proofs #6686

stabilization-equivalent-proofs #6686

AdoAdoAdo commented Dec 19, 2024 •

edited

Loading

sstanculeanu Jan 10, 2025

sstanculeanu Jan 10, 2025 •

edited

Loading

sstanculeanu Jan 10, 2025 •

edited

Loading

sstanculeanu Jan 10, 2025

AdoAdoAdo Jan 13, 2025

sstanculeanu Jan 10, 2025 •

edited

Loading

AdoAdoAdo Jan 14, 2025

sstanculeanu Jan 10, 2025

AdoAdoAdo Jan 14, 2025

sstanculeanu Jan 10, 2025

AdoAdoAdo Jan 20, 2025

ssd04 Jan 13, 2025

AdoAdoAdo Jan 14, 2025

ssd04 Jan 14, 2025

AdoAdoAdo Jan 14, 2025

sstanculeanu Jan 20, 2025

sstanculeanu Jan 20, 2025

ssd04 Jan 20, 2025

AdoAdoAdo Jan 20, 2025 •

edited

Loading

ssd04 Jan 20, 2025

AdoAdoAdo Jan 20, 2025

ssd04 Jan 20, 2025

	log.Error("SubroundsHandler.EpochStartAction: cannot initialize subrounds", "error", err)
	log.Error("SubroundsHandler.EpochConfirmed: cannot initialize subrounds", "error", err)

stabilization-equivalent-proofs #6686

stabilization-equivalent-proofs #6686

Conversation

AdoAdoAdo commented Dec 19, 2024 • edited Loading

Reasoning behind the pull request

Proposed changes

Testing procedure

Pre-requisites

Choose a reason for hiding this comment

sstanculeanu Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

sstanculeanu Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sstanculeanu Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AdoAdoAdo Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AdoAdoAdo commented Dec 19, 2024 •

edited

Loading

sstanculeanu Jan 10, 2025 •

edited

Loading

sstanculeanu Jan 10, 2025 •

edited

Loading

sstanculeanu Jan 10, 2025 •

edited

Loading

AdoAdoAdo Jan 20, 2025 •

edited

Loading