Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MultiNode Soak Testing #894

Merged
merged 266 commits into from
Nov 15, 2024
Merged

MultiNode Soak Testing #894

merged 266 commits into from
Nov 15, 2024

Conversation

DylanTinianov
Copy link
Contributor

@DylanTinianov DylanTinianov commented Oct 17, 2024 •
edited
Loading

Changes

  • Enabled the use of Multiple RPCs for Soak Testing
  • Added run_soak_test.sh to easily trigger soak tests and exit the terminal at the correct time
  • Wrapped SendTransaction() functionality within txSender.IfStarted(func() {...})
  • Improved TransactionSender context handling
  • Slightly relaxed some default configurations
  • Added reader getters to ensure a single RPC is not passed around negating MultiNode

Closes: https://smartcontract-it.atlassian.net/browse/NONEVM-842

Latest Soak Test (Devnet)

Soak tests are being run using a local set up.

CommitHash ca7c982 (latest)
Namespace: solana-gauntlet-embedded-d7efd
Num RPCs Latest Test SyncThreshold
Grafana Logs

Config

# This is the default configuration so OCR2 tests can run without issues
[ChainlinkImage]
version=<Latest CI Build Image>

[Logging]
test_log_collect=false

[Logging.LogStream]
log_targets=["file"]
log_producer_timeout="10s"
log_producer_retry_limit=10

[Common]
internal_docker_repo = ***
inside_k8 = true
network = "devnet"
user = "DylanTinianov"
stateful_db = true
devnet_image = "layerzerolabs/solana:1.17.31"

[OCR2]
node_count = 6
test_duration = "72h"
number_of_rounds = 3

RPCs

Using 3 internal solana-devnet RPCs

DylanTinianov and others added 30 commits September 6, 2024 14:58
@DylanTinianov
Add defaults
95e1c9a
@DylanTinianov
Add latest block methods
45a7596
@DylanTinianov
Merge branch 'develop' into BCI-2835-integrate-multinode
6641bc9
@DylanTinianov
Address comments
d8d312c
@DylanTinianov
Merge branch 'develop' into BCI-2835-integrate-multinode
aa3c068
@DylanTinianov
lint
3c3756e
@DylanTinianov
Merge branch 'develop' into BCI-2835-integrate-multinode
7c8b55d
@DylanTinianov
Fix lint overflow issues
2521670
@DylanTinianov
Update transaction_sender.go
5b5cfd6
@DylanTinianov
Fix lint
690f812
@DylanTinianov
Validate node config
fd3823b
@DylanTinianov
Update toml.go
4bf96b7
@DylanTinianov
Add SendOnly nodes
c1b83a5
@DylanTinianov
Merge branch 'BCI-2835-integrate-multinode' into BCFR-302-multinode-toml
c58dbc8
@DylanTinianov
Merge branch 'BCFR-302-multinode-toml' into BCFR-298-multinode-client
d90e3b8
@DylanTinianov
Use pointers on config
e0231b0
@DylanTinianov
Fix conflicts
0122a5f
@DylanTinianov
Add test outlines
8b13548
@DylanTinianov
Use test context
8aa39f6
@DylanTinianov
Merge branch 'BCFR-302-multinode-toml' into BCFR-298-multinode-client
94f3669
@DylanTinianov
Merge branch 'BCI-2835-integrate-multinode' into BCFR-302-multinode-toml
0e5a35e
@DylanTinianov
Merge branch 'BCFR-302-multinode-toml' into BCFR-298-multinode-client
06bf565
@DylanTinianov
Use configured selection mode
b5ff16d
@DylanTinianov
Set defaults
1b3a101
@DylanTinianov
Merge branch 'develop' into BCFR-302-multinode-toml
b7cc350
@DylanTinianov
lint
0afd8da
@DylanTinianov
Add nil check
b99b90c
@DylanTinianov
Merge branch 'BCFR-302-multinode-toml' into BCFR-298-multinode-client
8080fed
@DylanTinianov
Add client test
05524a1
@DylanTinianov
Add subscription test
6b92e70
aalu1418
aalu1418 requested changes Nov 15, 2024
View reviewed changes
Copy link
Collaborator

@aalu1418 aalu1418 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall looks good - just some code cleanliness things

pkg/solana/client/multinode/poller.go Outdated Show resolved Hide resolved
pkg/solana/config_tracker.go Outdated Show resolved Hide resolved
pkg/solana/cache_test.go Outdated Show resolved Hide resolved
pkg/solana/relay.go Outdated Show resolved Hide resolved
pkg/solana/state_cache.go Outdated Show resolved Hide resolved
pkg/solana/transmissions_cache.go Outdated Show resolved Hide resolved
dhaidashenko
dhaidashenko previously approved these changes Nov 15, 2024
View reviewed changes
aalu1418
aalu1418 requested changes Nov 15, 2024
View reviewed changes
pkg/solana/state_cache.go Outdated Show resolved Hide resolved
@aalu1418 aalu1418 enabled auto-merge (squash) November 15, 2024 18:26
@cl-sonarqube-production
Copy link

Quality Gate failed Quality Gate failed

Failed conditions
62.0% Coverage on New Code (required ≥ 75%)

See analysis details on SonarQube

@aalu1418 aalu1418 merged commit 8b8369c into develop Nov 15, 2024
34 of 35 checks passed
@aalu1418 aalu1418 deleted the BCFR-622-multinode-soak-testing branch November 15, 2024 19:11
dhaidashenko added a commit that referenced this pull request Dec 20, 2024
@DylanTinianov @smickovskid @dhaidashenko
MultiNode Soak Testing (#894)
115f36c 
* Add defaults

* Add latest block methods

* Address comments

* lint

* Fix lint overflow issues

* Update transaction_sender.go

* Fix lint

* Validate node config

* Update toml.go

* Add SendOnly nodes

* Use pointers on config

* Add test outlines

* Use test context

* Use configured selection mode

* Set defaults

* lint

* Add nil check

* Add client test

* Add subscription test

* tidy

* Fix imports

* Update chain_test.go

* Update multinode.go

* Add comments

* Update multinode.go

* Wrap multinode config

* Fix imports

* Update .golangci.yml

* Use MultiNode

* Add multinode to txm

* Use MultiNode

* Update chain.go

* Update balance_test.go

* Add retries

* Fix head

* Update client.go

* lint

* lint

* Use MultiNode TxSender

* Update txm_internal_test.go

* Address comments

* Remove total difficulty

* Register polling subs

* Extract MultiNodeClient

* Remove caching changes

* Undo cache changes

* Fix tests

* Update chain.go

* Fix variables

* Move classify errors

* Fix imports

* lint

* Update txm_internal_test.go

* Update txm_internal_test.go

* lint

* Fix error classification

* Update txm_internal_test.go

* Update multinode_client.go

* lint

* Update classify_errors.go

* Update classify_errors.go

* Add tests

* Add test coverage

* lint

* Add dial comment

* CTF bump for image build

* Update pkg/solana/client/multinode_client.go

Co-authored-by: Dmytro Haidashenko <34754799+dhaidashenko@users.noreply.github.com>

* Update txm.go

* Create loader

* Update transaction_sender.go

* Fix tests

* Update txm_internal_test.go

* lint

* Update txm.go

* Add ctx

* Fix imports

* Add SendTxResult to TxSender

* Update chain_test.go

* Enable MultiNode

* Move error classification

* Add MultiNode config

* Use loader

* Update multinode.go

* Update multinode.go

* Use loader in txm tests

* lint

* Update testconfig.go

* Update loader

* Use single RPC

* Fix tests

* lint

* Use default thresholds

* Address comments

* Update classify_errors.go

* Update testconfig.go

* Update errors

* lint

* Fix SendTransaction

* Update chain.go

* Update sendTx

* Fix ctx issues

* Enable multiple RPCs in soak tests

* Update defaults for testing

* Add health check tags

* Increase sync threshold

* Validate heads

* Use latestChainInfo

* Fix AliveLoop bug

* Update configurations

* Update transaction_sender.go

* Get chain info

* Update ctx

* Update transaction_sender.go

* Update transaction_sender.go

* Increase tx timeout

* Update transaction_sender.go

* Update ctx

* Add timer

* Update transaction_sender.go

* Update transaction_sender.go

* Update testconfig.go

* Fix ctx

* Remove debug logging

* Update run_soak_test.sh

* lint

* Add debugging logs

* Fix ctx cancel

* Fix ctx cancel

* Fix DoAll ctx

* Remove debugging logs

* Remove logs

* defer reportWg

* Add result ctx logging

* log on close

* Update transaction_sender.go

* add cancel func

* Update transaction_sender.go

* Update transaction_sender.go

* Add ctx to reportSendTxAnomalies

* Update comments

* Fix comments

* Address comments

* lint

* lint

* Pass context

* Update node_lifecycle.go

* Use get reader function

* Make rpcurls plural

* Fix reader getters

* lint

* fix imports

* Update transaction_sender.go

* Remove TxError

* Rename getReader

* lint

* Update chain_test.go

* Update transmissions_cache.go

* Update run_soak_test.sh

* Fix deprecated method

* Clean up getReader

* Use AccountReader

---------

Co-authored-by: Damjan Smickovski <smickovskid@outlook.com>
Co-authored-by: Dmytro Haidashenko <34754799+dhaidashenko@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aalu1418 aalu1418 aalu1418 approved these changes

@dhaidashenko dhaidashenko dhaidashenko approved these changes

@skippaDaBitFlippa skippaDaBitFlippa skippaDaBitFlippa approved these changes

@jmank88 jmank88 Awaiting requested review from jmank88

Assignees

@DylanTinianov DylanTinianov

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@DylanTinianov @jmank88 @skippaDaBitFlippa @dhaidashenko @aalu1418 @smickovskid