Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat/balance data tries #4636

Merged
merged 143 commits into from
May 30, 2023
Merged

Feat/balance data tries #4636

merged 143 commits into from
May 30, 2023

Conversation

BeniaminDrasovean
Copy link
Contributor

@BeniaminDrasovean BeniaminDrasovean commented Oct 24, 2022 •
edited
Loading

Reasoning behind the pull request

  • The keys at which values are saved in a data trie are not random. Because of this, the data tries are not balanced, thus resulting in more intermediary nodes.

Proposed changes

  • When saving some data in the data trie, do not save at key but rather at hash(key). This way, the keys will be random, resulting in balanced data tries.
  • In order for this change to be backwards compatible, add Version field to trie nodes. Nodes that are accessed will be automatically migrated to the new version (where the storage key is hash(key))
  • Add a built in function that will migrate data trie nodes when it is called
  • Add a new API endpoint which will return true if the data trie of a specified account is fully migrated.

Testing procedure

  • Deploy a contract with some data before the activation of this feature. After the activation, access all the keys and check that they are automatically migrated.
  • Deploy a contract with some data before the activation of this feature. Use the built in function to migrate the data trie after the feature is active.
  • Check that the data that is added after the feature is active is saved in the trie by using the new version.

Pre-requisites

Based on the Contributing Guidelines the PR author and the reviewers must check the following requirements are met:

  • was the PR targeted to the correct branch?
  • if this is a larger feature that probably needs more than one PR, is there a feat branch created?
  • if this is a feat branch merging, do all satellite projects have a proper tag inside go.mod?

BeniaminDrasovean and others added 30 commits October 24, 2022 14:51
@BeniaminDrasovean
add hasher to trackable data trie
42a73c8
@BeniaminDrasovean
sort imports
498b5f7
@BeniaminDrasovean
Merge pull request #4633 from ElrondNetwork/add-hasher-to-trackable-d…
c279ee4 
…ata-trie

Add hasher to trackable data trie
@BeniaminDrasovean
add marshalizer to trackable data trie
9466b97
@BeniaminDrasovean
sort imports
9bc791f
@BeniaminDrasovean
fix after review
7e51156
@BeniaminDrasovean
Merge pull request #4640 from ElrondNetwork/add-marshaller-to-data-tr…
535dd32 
…ie-tracker

Add marshaller to data trie tracker
@BeniaminDrasovean
Merge remote-tracking branch 'origin/rc/v1.4.0' into 28.10-merge-rc/1…
8f81258 
….4.0-to-feat/balance-data-tries

# Conflicts:
#	integrationTests/testInitializer.go
#	integrationTests/vm/testInitializer.go
#	state/trackableDataTrie_test.go
@BeniaminDrasovean
Merge pull request #4652 from ElrondNetwork/28.10-merge-rc/1.4.0-to-f…
fc8ec3e 
…eat/balance-data-tries

28.10 merge rc/1.4.0 to feat/balance data tries
@BeniaminDrasovean
add AutoBalanceDataTriesEnableEpoch flag
ebf7e05
@BeniaminDrasovean
propagate enableEpochsHandler to userAccount
e25cfed
@BeniaminDrasovean
generate dataTrieValue based on proto file
a613e1c
@BeniaminDrasovean
generate a struct for dataTrieValue and pass a trieLeafParser to GetA…
12f8d59 
…llLeavesOnChannel
@BeniaminDrasovean
fix after review
39467c0
@BeniaminDrasovean
refactor comment for NewPeerAccount
33c763f
@BeniaminDrasovean
Merge pull request #4680 from ElrondNetwork/enable-epochs-handler-to-…
02f9e7d 
…data-trie

Enable epochs handler to data trie
@BeniaminDrasovean
Merge branch 'feat/balance-data-tries' into data-trie-value
1bc5e2e
@BeniaminDrasovean
rename disabledTrieLeafParser to trieLeafParserV1
e48cace
@BeniaminDrasovean
Merge pull request #4686 from ElrondNetwork/data-trie-value
4daa25f 
Data trie value
@BeniaminDrasovean
implement new version of data trie value storage
e035b91
@BeniaminDrasovean
Merge branch 'feat/balance-data-tries' into hash-keys-for-data-tries
6667f05 
# Conflicts:
#	epochStart/metachain/systemSCs.go
#	integrationTests/testProcessorNode.go
#	node/node.go
#	node/trieIterators/delegatedListProcessor.go
#	node/trieIterators/directStakedListProcessor.go
#	node/trieIterators/stakeValuesProcessor.go
@BeniaminDrasovean
rename trieLeafParser to trieLeafParserV2
67b99f2
@BeniaminDrasovean
sort imports
fd6d2c3
@BeniaminDrasovean
fix after review
36bbcb2
@BeniaminDrasovean
replace NewEmptyUserAccount with NewUserAccountFromBytes
a668bf0
@BeniaminDrasovean
fix edgecase with nil val on SaveKeyVal()
a6e5367
@BeniaminDrasovean
Merge pull request #4695 from ElrondNetwork/hash-keys-for-data-tries
d30b794 
Hash keys for data tries
@BeniaminDrasovean
add key size to total leaf size computation only after autobalancing …
a041022 
…enabled
@BeniaminDrasovean
remove key from maxLeafSize computation
7d267e9
@BeniaminDrasovean
Merge pull request #4729 from ElrondNetwork/fix-leaf-size-check
32a03f4 
remove key size from total leaf size computation
BeniaminDrasovean and others added 8 commits May 23, 2023 10:25
@BeniaminDrasovean
Merge remote-tracking branch 'origin/rc/v1.6.0' into merge-rc/v1.6.0-…
ac7093f 
…in-feat/balance-data-tries-23may

# Conflicts:
#	api/errors/errors.go
#	api/mock/facadeStub.go
#	common/enablers/enableEpochsHandler.go
#	common/enablers/enableEpochsHandler_test.go
#	common/enablers/epochFlags.go
#	common/interface.go
#	config/epochConfig.go
#	errors/errors.go
#	facade/initial/initialNodeFacade_test.go
#	facade/mock/nodeStub.go
#	factory/processing/processComponents.go
#	factory/statusCore/statusCoreComponents_test.go
#	go.mod
#	go.sum
#	integrationTests/interface.go
#	outport/factory/notifierFactory_test.go
#	outport/notifier/eventNotifier_test.go
#	sharding/mock/enableEpochsHandlerMock.go
#	testscommon/components/default.go
#	testscommon/enableEpochsHandlerMock/enableEpochsHandlerStub.go
#	trie/trieStorageManager_test.go
#	vm/systemSmartContracts/governance_test.go
#	vm/systemSmartContracts/validator_test.go
@BeniaminDrasovean
fix after merge
db120b1
@BeniaminDrasovean
Merge pull request #5268 from multiversx/merge-rc/v1.6.0-in-feat/bala…
6654e8d 
…nce-data-tries-23may

Merge rc/v1.6.0 in feat/balance data tries 23may
@BeniaminDrasovean
Merge remote-tracking branch 'origin/rc/v1.6.0' into merge-rc/v1.6.0-…
63feb40 
…in-feat/balance-data-tries-24may

# Conflicts:
#	dataRetriever/factory/storageRequestersContainer/baseRequestersContainerFactory.go
#	dataRetriever/factory/storageRequestersContainer/metaRequestersContainerFactory.go
#	dataRetriever/factory/storageRequestersContainer/shardRequestersContainerFactory.go
#	epochStart/metachain/baseRewards_test.go
#	epochStart/metachain/systemSCs_test.go
#	errors/errors.go
#	factory/processing/blockProcessorCreator_test.go
#	factory/state/stateComponents.go
#	genesis/process/genesisBlockCreator.go
#	genesis/process/genesisBlockCreator_test.go
#	go.mod
#	go.sum
#	integrationTests/benchmarks/loadFromTrie_test.go
#	integrationTests/state/stateTrieClose/stateTrieClose_test.go
#	integrationTests/testProcessorNode.go
#	integrationTests/vm/testInitializer.go
#	state/accountsDB.go
#	state/syncer/errors.go
#	testscommon/components/components.go
#	trie/branchNode.go
#	trie/branchNode_test.go
#	trie/doubleListSync_test.go
#	trie/errors.go
#	trie/extensionNode.go
#	trie/extensionNode_test.go
#	trie/factory/trieCreator.go
#	trie/factory/trieCreator_test.go
#	trie/interface.go
#	trie/leafNode.go
#	trie/patriciaMerkleTrie.go
#	trie/patriciaMerkleTrie_test.go
#	trie/trieStorageManager_test.go
#	update/genesis/import.go
@BeniaminDrasovean
fix after merge
f10b18c
@BeniaminDrasovean
Merge remote-tracking branch 'origin/rc/v1.6.0' into merge-rc/v1.6.0-…
1c57589 
…in-feat/balance-data-tries-24may

# Conflicts:
#	go.mod
#	go.sum
@BeniaminDrasovean
update go mod
e4e66c3
@BeniaminDrasovean
Merge pull request #5279 from multiversx/merge-rc/v1.6.0-in-feat/bala…
1f61f5d 
…nce-data-tries-24may

Merge rc/v1.6.0 in feat/balance data tries 24may
BeniaminDrasovean and others added 9 commits May 25, 2023 14:09
@BeniaminDrasovean
change epoch in which autoBalanceDataTries is enabled
b128a5c
@BeniaminDrasovean
Merge pull request #5282 from multiversx/change-auto-balance-enable-e…
d1cbee2 
…poch

change epoch in which autoBalanceDataTries is enabled
@BeniaminDrasovean
Merge remote-tracking branch 'origin/rc/v1.6.0' into update-feat/bala…
d19560a 
…nce-data-tries-29may

# Conflicts:
#	common/enablers/enableEpochsHandler.go
#	common/enablers/epochFlags.go
#	common/interface.go
#	go.mod
#	go.sum
#	sharding/mock/enableEpochsHandlerMock.go
#	testscommon/enableEpochsHandlerMock/enableEpochsHandlerStub.go
@BeniaminDrasovean
fix after merge
19dd063
@BeniaminDrasovean
Merge pull request #5287 from multiversx/update-feat/balance-data-tri…
876f173 
…es-29may

Update feat/balance data tries 29may
@BeniaminDrasovean
update go mod
838969c
@BeniaminDrasovean
Merge pull request #5292 from multiversx/integrate-latest-changes-in-…
056fd44 
…feat/balance-data-tries

update go mod
@BeniaminDrasovean
add proper releases
11e0647
@BeniaminDrasovean
Merge pull request #5298 from multiversx/balance-data-tries-proper-re…
c631b80 
…leases

add proper releases
@BeniaminDrasovean BeniaminDrasovean marked this pull request as ready for review May 30, 2023 09:36
gabi-vuls
gabi-vuls approved these changes May 30, 2023
View reviewed changes
Copy link
Contributor

@gabi-vuls gabi-vuls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

System test passed

@BeniaminDrasovean BeniaminDrasovean merged commit 37f967b into rc/v1.6.0 May 30, 2023
@BeniaminDrasovean BeniaminDrasovean deleted the feat/balance-data-tries branch May 30, 2023 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gabi-vuls gabi-vuls gabi-vuls approved these changes

@iulianpascalau iulianpascalau iulianpascalau approved these changes

@ssd04 ssd04 ssd04 approved these changes

@sasurobert sasurobert sasurobert approved these changes

Assignees
No one assigned
Labels
area:trie complexity:high impact:high interested:protocol size:large type:feature New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@BeniaminDrasovean @sasurobert @ssd04 @iulianpascalau @gabi-vuls