Add subsystem benchmarks for `availability-distribution` and `biftield-distribution` (availability write) #2970

sandreim · 2024-01-17T17:27:01Z

Introduce a new test objective : DataAvailabilityWrite.

The new benchmark measures the network and cpu usage of availability-distribution, biftield-distribution and availability-store subsystems from the perspective of a validator node during the process when candidates are made available.

Additionally I refactored the networking emulation to support bandwidth acounting and limits of incoming and outgoing requests.

Screenshot of succesful run

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

…reim/subsystem-bench

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

…reim/availability-write-bench

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

alexggh

Some nits, looks good to me.

polkadot/node/subsystem-bench/Cargo.toml

polkadot/node/subsystem-bench/src/core/environment.rs

alindima

Overall LGTM, nice work!

polkadot/node/subsystem-bench/src/availability/mod.rs

alindima · 2024-01-23T09:41:40Z

polkadot/node/subsystem-bench/src/availability/mod.rs

+		state.backed_candidates.push(
+			candidate_hashes
+				.get(&Hash::repeat_byte(block_num as u8))
+				.expect("just inserted above")
+				.get(0)
+				.expect("just inserted above")
+				.clone(),
+		);


I think we could just store the candidate hash in the backed_candidates map.

I am not sure I understand correctly, but backed_candidates is a vec ( per relay block candidates backed by the node under test) and it stores CandidateReceipt

right, in my mind vec is also a map from an integer to smth :D

i am suggesting based on my reading of the code, that we could only store the CandidateHash in the backed_candidates instead of the full receipt and then use the candidate index to fetch the receipt from the candidate_receipt_templates, if that makes sense.

polkadot/node/subsystem-bench/src/availability/mod.rs

polkadot/node/subsystem-bench/src/core/mock/network_bridge.rs

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

…reim/availability-write-bench Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

paritytech-cicd-pr · 2024-01-25T15:37:44Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5029861

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

#2970 accidentally added a crate twice to the workspace. Now extending the workspace check to explicitly error then. I think the check should also be required now. --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Polkadot-Forum · 2024-05-21T14:59:54Z

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/what-are-subsystem-benchmarks/8212/1

sandreim added 30 commits October 25, 2023 11:50

skeleton

01af630

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

wip

7c22abe

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

measure tput and fixes

c3adc77

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

add network emulation

31b0351

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

cleanup

e4bb037

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Add latency emulation

a694924

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

support multiple pov sizes

7ca4dba

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

new metric in recovery and more testing

0430b5b

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

CLI update and fixes

027bcd8

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

peer stats

5a05da0

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Switch stats to atomics

895e8d6

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

add more network metrics, new load generator

a2fb0c9

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

refactor

d1b9fa3

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

pretty cli + minor refactor + remove unused

c5937ab

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

update

d6c259d

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

remove comment

050529b

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

separate cli options for availability

cb38be5

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

implement unified and extensible configuration

24a736a

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Prepare to swtich to overseer

2843865

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Merge branch 'master' of github.com:paritytech/polkadot-sdk into sand…

fd4620e

…reim/subsystem-bench

add mocked subsystems

b17a147

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

full overseer based implementation complete

4724d8c

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

make clean

7aed30f

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

more cleaning

b51485b

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

more cleaning

7e46444

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

proper overseer control

d3df927

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

refactor CLI display of env stats

7557768

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Add grafana dashboards for DA read

787dc00

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

network stats fixes

cd18f8d

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

move examples and grafana

e8506b3

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim added 2 commits January 17, 2024 19:10

fix print

8c00d70

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Fix block time

f839f77

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim added T0-node This PR/Issue is related to the topic “node”. T10-tests This PR/Issue is related to tests. T12-benchmarks This PR/Issue is related to benchmarking and weights. labels Jan 17, 2024

sandreim added 3 commits January 17, 2024 19:31

Merge branch 'master' of github.com:paritytech/polkadot-sdk into sand…

b6807f2

…reim/availability-write-bench

clippy

48508ba

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Fix ASCII art

28d9f77

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

alexggh approved these changes Jan 19, 2024

View reviewed changes

polkadot/node/subsystem-bench/Cargo.toml Outdated Show resolved Hide resolved

polkadot/node/subsystem-bench/src/core/environment.rs Outdated Show resolved Hide resolved

AndreiEres approved these changes Jan 22, 2024

View reviewed changes

alindima reviewed Jan 23, 2024

View reviewed changes

alindima mentioned this pull request Jan 23, 2024

Introduce approval-voting/distribution benchmark #2621

Merged

sandreim added 3 commits January 23, 2024 13:56

feedback 1/2

d8523a9

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

feedback 2/2

d9c685d

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

nits

bec7998

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

alindima approved these changes Jan 23, 2024

View reviewed changes

Merge branch 'master' of github.com:paritytech/polkadot-sdk into sand…

848b54f

…reim/availability-write-bench Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim enabled auto-merge January 25, 2024 15:32

sandreim added 3 commits January 25, 2024 18:18

merge damage

1614ce8

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

add prdoc

32d5d07

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

fix md and cargo lock

68fbd6e

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim added this pull request to the merge queue Jan 25, 2024

Merged via the queue into master with commit 47e46d1 Jan 25, 2024
122 of 125 checks passed

sandreim deleted the sandreim/availability-write-bench branch January 25, 2024 17:52

ggwpez mentioned this pull request Jan 26, 2024

Error on duplicated crates #3082

Merged

github-actions bot mentioned this pull request Mar 13, 2024

Update polkadot-sdk from v1.3.0 to v1.7.2 moonbeam-foundation/moonbeam#2703

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add subsystem benchmarks for `availability-distribution` and `biftield-distribution` (availability write) #2970

Add subsystem benchmarks for `availability-distribution` and `biftield-distribution` (availability write) #2970

sandreim commented Jan 17, 2024 •

edited

Loading

alexggh left a comment

alindima left a comment

alindima Jan 23, 2024

sandreim Jan 23, 2024 •

edited

Loading

alindima Jan 23, 2024

paritytech-cicd-pr commented Jan 25, 2024

Polkadot-Forum commented May 21, 2024

Add subsystem benchmarks for availability-distribution and biftield-distribution (availability write) #2970

Add subsystem benchmarks for availability-distribution and biftield-distribution (availability write) #2970

Conversation

sandreim commented Jan 17, 2024 • edited Loading

alexggh left a comment

Choose a reason for hiding this comment

alindima left a comment

Choose a reason for hiding this comment

alindima Jan 23, 2024

Choose a reason for hiding this comment

sandreim Jan 23, 2024 • edited Loading

Choose a reason for hiding this comment

alindima Jan 23, 2024

Choose a reason for hiding this comment

paritytech-cicd-pr commented Jan 25, 2024

Polkadot-Forum commented May 21, 2024

Add subsystem benchmarks for `availability-distribution` and `biftield-distribution` (availability write) #2970

Add subsystem benchmarks for `availability-distribution` and `biftield-distribution` (availability write) #2970

sandreim commented Jan 17, 2024 •

edited

Loading

sandreim Jan 23, 2024 •

edited

Loading