Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: add store seed for stress test #621

Open
wants to merge 36 commits into
base: next
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
a7d4130
test: add stress binary
TomasArrachea Jan 15, 2025
a2cdcd8
test: set 16 batches per block and 16 txs per batch
TomasArrachea Jan 15, 2025
afec921
Merge branch 'next' into tomasarrachea-stress-test
TomasArrachea Jan 16, 2025
387d95d
test: use only block builder instead of block producer
TomasArrachea Jan 16, 2025
4395203
test: parallelize account grinding
TomasArrachea Jan 16, 2025
dc3b1a2
test: add sync request test
TomasArrachea Jan 17, 2025
63916b8
feat: add cli to load store
TomasArrachea Jan 17, 2025
5c39a4c
Merge branch 'next' into tomasarrachea-stress-test
TomasArrachea Jan 17, 2025
1536920
refactor: move code to helper functions
TomasArrachea Jan 20, 2025
bf64e2c
fix: use different init seed on each account
TomasArrachea Jan 20, 2025
5afac9f
fix: use genesis block as anchor
TomasArrachea Jan 20, 2025
f8fcf22
fix: undo StoreClient as pub
TomasArrachea Jan 20, 2025
b22ef0d
fix: remove miden-objects testing feature
TomasArrachea Jan 20, 2025
77d73b4
docs: add block-producer features documentation
TomasArrachea Jan 20, 2025
ee0987c
feat: add makefile target for miden-stress-test
TomasArrachea Jan 20, 2025
ccdf645
wip: add authenticated notes
TomasArrachea Jan 20, 2025
5aed8d9
chore: rename subcrate to miden-node-stress-test
TomasArrachea Jan 21, 2025
a6d4c1a
chore: alphabetize dependencies and drop patch version
TomasArrachea Jan 21, 2025
018078c
chore: update README
TomasArrachea Jan 21, 2025
46a976f
fix: rename num_accounts parameter
TomasArrachea Jan 21, 2025
2e2e164
Merge branch 'next' into tomasarrachea-stress-test
TomasArrachea Jan 21, 2025
f009a8a
feat: refactor into separate helper functions
TomasArrachea Jan 21, 2025
08478dc
Merge branch 'next' into tomasarrachea-stress-test
TomasArrachea Jan 21, 2025
348b1bb
feat: remove query for note inclusion proof
TomasArrachea Jan 21, 2025
dc7e265
fix: avoid blocking tokio runtime
TomasArrachea Jan 23, 2025
661c2cb
Merge branch 'next' into tomasarrachea-stress-test
TomasArrachea Jan 23, 2025
00e86df
fix: format
TomasArrachea Jan 23, 2025
046f690
feat: use unbounded channels
TomasArrachea Jan 23, 2025
e545d92
fix: add docs and move seed_store definition
TomasArrachea Jan 23, 2025
606feb1
fix: alphabetize dependencies and declare as dev
TomasArrachea Jan 23, 2025
886554d
fix: update makefile target
TomasArrachea Jan 23, 2025
4f63c1a
Merge branch 'next' into tomasarrachea-stress-test
SantiagoPittella Jan 28, 2025
0820ffd
review: update crate description
SantiagoPittella Jan 28, 2025
81b1d04
review: fix block-producer readme
SantiagoPittella Jan 28, 2025
d1ca045
review: make testing deps optional for block-producer
SantiagoPittella Jan 28, 2025
28f0632
review: add additional metrics
SantiagoPittella Jan 28, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
members = [
"bin/node",
"bin/faucet",
"bin/stress-test",
"crates/block-producer",
"crates/proto",
"crates/rpc-proto",
Expand Down
33 changes: 33 additions & 0 deletions bin/stress-test/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
[package]
name = "miden-stress-test"
bobbinth marked this conversation as resolved.
Show resolved Hide resolved
edition.workspace = true
rust-version.workspace = true
version.workspace = true
license.workspace = true
authors.workspace = true
homepage.workspace = true
repository.workspace = true
exclude.workspace = true
readme.workspace = true

[dependencies]
async-trait = { version = "0.1" }
anyhow = "1.0"
itertools = { version = "0.13" }
miden-lib = { workspace = true }
miden-node-proto = { workspace = true }
miden-node-utils = { workspace = true }
miden-objects = { workspace = true }
miden-processor = { workspace = true }
miden-stdlib = { workspace = true }
miden-tx = { workspace = true }
miden-node-block-producer = { workspace = true }
miden-node-store = { workspace = true }
rand = { version = "0.8" }
serde = { version = "1.0", features = ["derive"] }
thiserror = { workspace = true }
tokio = { workspace = true, features = ["rt-multi-thread", "net", "macros", "sync", "time"] }
tokio-stream = { workspace = true, features = ["net"] }
tonic = { workspace = true }
tracing = { workspace = true }
rand_chacha = "0.3"
117 changes: 117 additions & 0 deletions bin/stress-test/src/main.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
use anyhow::Context;
use miden_lib::{
accounts::{auth::RpoFalcon512, faucets::BasicFungibleFaucet, wallets::BasicWallet},
transaction::TransactionKernel,
};
use miden_node_block_producer::{
batch_builder::TransactionBatch, config::BlockProducerConfig, server::BlockProducer,
test_utils::MockProvenTxBuilder,
};
use miden_node_store::{config::StoreConfig, server::Store};
use miden_objects::{
accounts::{AccountBuilder, AccountIdAnchor, AccountStorageMode, AccountType},
assets::{Asset, FungibleAsset, TokenSymbol},
crypto::dsa::rpo_falcon512::SecretKey,
testing::notes::NoteBuilder,
transaction::OutputNote,
Digest, Felt,
};
use miden_processor::crypto::RpoRandomCoin;
use rand::Rng;
use tokio::task::JoinSet;

#[tokio::main]
async fn main() {
let block_producer_config = BlockProducerConfig::default();
let store_config = StoreConfig::default();
let mut join_set = JoinSet::new();

// Start store
let store = Store::init(store_config).await.context("Loading store").unwrap();
let _ = join_set.spawn(async move { store.serve().await.context("Serving store") }).id();

// Start block-producer
// TODO: is the full the BlockProducer needed? we should instantiate only a BlockBuilder
let block_producer = BlockProducer::init(block_producer_config)
.await
.context("Loading block-producer")
.unwrap();
bobbinth marked this conversation as resolved.
Show resolved Hide resolved

println!("Creating new faucet account...");
let coin_seed: [u64; 4] = rand::thread_rng().gen();
let mut rng = RpoRandomCoin::new(coin_seed.map(Felt::new));
let key_pair = SecretKey::with_rng(&mut rng);
let init_seed = [0_u8; 32];
let (new_faucet, _seed) = AccountBuilder::new()
.init_seed(init_seed)
.anchor(AccountIdAnchor::PRE_GENESIS)
.account_type(AccountType::FungibleFaucet)
.storage_mode(AccountStorageMode::Private)
.with_component(RpoFalcon512::new(key_pair.public_key()))
.with_component(
BasicFungibleFaucet::new(TokenSymbol::new("TEST").unwrap(), 2, Felt::new(100_000))
.unwrap(),
)
.build()
.unwrap();
let faucet_id = new_faucet.id();

// The amount of blocks to create and process.
const BATCHES_PER_BLOCK: usize = 4;
const N_BLOCKS: usize = 250_000;
// 250_000 blocks * 4 batches/block * 1 accounts/batch = 1_000_000 accounts
// Each batch contains 2 txs: one to create a note and another to consume it.

for block_num in 0..N_BLOCKS {
let mut batches = Vec::with_capacity(BATCHES_PER_BLOCK);
for _ in 0..BATCHES_PER_BLOCK {
// Create wallet
let (new_account, _) = AccountBuilder::new()
.init_seed(init_seed)
.anchor(AccountIdAnchor::PRE_GENESIS)
.account_type(AccountType::RegularAccountImmutableCode)
.storage_mode(AccountStorageMode::Private)
.with_component(RpoFalcon512::new(key_pair.public_key()))
.with_component(BasicWallet)
.build()
.unwrap();
let account_id = new_account.id();

// Create note
let asset = Asset::Fungible(FungibleAsset::new(faucet_id, 10).unwrap());
let coin_seed: [u64; 4] = rand::thread_rng().gen();
let rng: RpoRandomCoin = RpoRandomCoin::new(coin_seed.map(Felt::new));
let note = NoteBuilder::new(faucet_id, rng)
.add_assets(vec![asset.clone()])
.build(&TransactionKernel::assembler())
.unwrap();

let batch = {
// First tx: create the note
let create_notes_tx = MockProvenTxBuilder::with_account(
faucet_id,
Digest::default(),
Digest::default(),
)
.output_notes(vec![OutputNote::Full(note.clone())])
.build();

// Second tx: consume the note
let consume_notes_txs = MockProvenTxBuilder::with_account(
account_id,
Digest::default(),
Digest::default(),
)
.unauthenticated_notes(vec![note])
.build();

TransactionBatch::new([&create_notes_tx, &consume_notes_txs], Default::default())
.unwrap()
};
batches.push(batch);
}
println!("Building block {}...", block_num);
// Inserts the block into the store sending it via StoreClient (RPC)
block_producer.block_builder.build_block(&batches).await.unwrap();
}
}
15 changes: 6 additions & 9 deletions crates/block-producer/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -17,29 +17,26 @@ tracing-forest = ["miden-node-utils/tracing-forest"]
[dependencies]
async-trait = { version = "0.1" }
itertools = { version = "0.13" }
miden-lib = { workspace = true }
miden-node-proto = { workspace = true }
miden-node-utils = { workspace = true }
miden-objects = { workspace = true }
miden-processor = { workspace = true }
miden-stdlib = { workspace = true }
miden-tx = { workspace = true }
rand = { version = "0.8" }
serde = { version = "1.0", features = ["derive"] }
thiserror = { workspace = true }
tokio = { workspace = true, features = ["rt-multi-thread", "net", "macros", "sync", "time"] }
tokio-stream = { workspace = true, features = ["net"] }
tonic = { workspace = true }
tracing = { workspace = true }

[dev-dependencies]
assert_matches = { workspace = true}
miden-air = { workspace = true }
winterfell = { version = "0.10" }
miden-objects = { workspace = true, features = ["testing"] }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that we should not be using "testing" in the dependency, can we use the feature only the dev-dependencies?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes!

rand_chacha = { version = "0.3", default-features = false }
miden-lib = { workspace = true, features = ["testing"] }
miden-node-test-macro = { path = "../test-macro" }
miden-objects = { workspace = true, features = ["testing"] }
miden-tx = { workspace = true, features = ["testing"] }

[dev-dependencies]
assert_matches = { workspace = true}
pretty_assertions = "1.4"
rand_chacha = { version = "0.3", default-features = false }
tokio = { workspace = true, features = ["test-util"] }
winterfell = { version = "0.10" }
1 change: 0 additions & 1 deletion crates/block-producer/src/batch_builder/batch.rs
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,6 @@ impl TransactionBatch {

/// Returns an iterator over (account_id, init_state_hash) tuples for accounts that were
/// modified in this transaction batch.
#[cfg(test)]
pub fn account_initial_states(&self) -> impl Iterator<Item = (AccountId, Digest)> + '_ {
self.updated_accounts
.iter()
Expand Down
2 changes: 1 addition & 1 deletion crates/block-producer/src/block_builder/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ impl BlockBuilder {
}

#[instrument(target = COMPONENT, skip_all, err)]
async fn build_block(&self, batches: &[TransactionBatch]) -> Result<(), BuildBlockError> {
pub async fn build_block(&self, batches: &[TransactionBatch]) -> Result<(), BuildBlockError> {
info!(
target: COMPONENT,
num_batches = batches.len(),
Expand Down
1 change: 0 additions & 1 deletion crates/block-producer/src/domain/transaction.rs
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,6 @@ impl AuthenticatedTransaction {
}
}

#[cfg(test)]
impl AuthenticatedTransaction {
//! Builder methods intended for easier test setup.

Expand Down
5 changes: 2 additions & 3 deletions crates/block-producer/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,10 @@ use std::time::Duration;

use mempool::BlockNumber;

#[cfg(test)]
pub mod test_utils;

mod batch_builder;
mod block_builder;
pub mod batch_builder;
pub mod block_builder;
mod domain;
mod errors;
mod mempool;
Expand Down
2 changes: 1 addition & 1 deletion crates/block-producer/src/server.rs
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ use crate::{
/// components.
pub struct BlockProducer {
batch_builder: BatchBuilder,
block_builder: BlockBuilder,
pub block_builder: BlockBuilder,
batch_budget: BatchBudget,
block_budget: BlockBudget,
state_retention: usize,
Expand Down
Loading