Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

safekeeper: add WAL ingestion benchmarks with pageserver fan-out #9569

Closed
Tracked by #9329
erikgrinaker opened this issue Oct 29, 2024 · 0 comments · Fixed by #9591
Closed
Tracked by #9329

safekeeper: add WAL ingestion benchmarks with pageserver fan-out #9569

erikgrinaker opened this issue Oct 29, 2024 · 0 comments · Fixed by #9591
Assignees
Labels
a/benchmark Area: related to benchmarking c/storage Component: storage

Comments

@erikgrinaker
Copy link
Contributor

For sharded ingest performance, we should add a benchmark that ingests WAL into a Safekeeper and fans it out into 8 shards across 8 pageservers.

Ideally, we would have a full end-to-end benchmark including compute, safekeeper cluster, and pageserver cluster with networking and S3. However, building this infrastructure is likely more work than we have resources for in this project.

For now, we'll write a Rust integration-style benchmark that runs the Safekeeper and Pageservers in memory on the local node, ignoring network latency, (possibly) disk latency, and compute overhead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a/benchmark Area: related to benchmarking c/storage Component: storage
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant