Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Epic: getpage@lsn benchmark #5771

Open
44 of 47 tasks
problame opened this issue Nov 2, 2023 · 2 comments
Open
44 of 47 tasks

Epic: getpage@lsn benchmark #5771

problame opened this issue Nov 2, 2023 · 2 comments
Assignees
@bayandin
Labels
c/storage/pageserver Component: storage: pageserver t/Epic Issue type: Epic

Comments

@problame
Copy link
Contributor

problame commented Nov 2, 2023 •
edited by jcsp
Loading

Motivation

https://www.notion.so/neondatabase/Test-Tools-633ddee0cf1d4e6b9962ff5df433a27d?pvs=4

DoD

There is a benchmark for getpage@lsn performance that does not require a compute to run for benchmark execution (ok for benchmark setup).

The benchmark is run as part of the performance regression tests

  • nightly against staging
  • as part of the pre-release testing

The benchmark results are reproducible for Pageserver developers.

Pageserver developers get alerted about perf regressions.

Tasks

High Level
Preview Give feedback

Impl
Preview Give feedback
  1. make CopyBothDuplex struct pub rust-postgres#25
  2. logging: support output to stderr #5896
  3. refer to our rust-postgres.git fork by branch name #5894
  4. pagectl: add subcommand to rewrite layer file summary #5933
  5. expose generation number in API #5934
  6. test suite: add method for generation-aware detachment of a tenant #5939
  7. concurrency-limit initial logical size calculation #5955
    problame
  8. add a Rust client for Pageserver management API #6127
  9. pageserver: add method to dump keyspace in mgmt api client #6145
  10. add a Rust client for Pageserver page_service #6128
  11. include timeline ids in tenant details response #6166
  12. add pageserver component-level benchmark (pagebench) #6174
  13. pagebench: always sends requests with latest=false #6209
    c/storage/pageserver t/bug
  14. pagebench: sends requests for block 0xffffffff on some relations #6210
    c/storage/pageserver t/bug
    problame
  15. memoization of repo_dir with the many tenants locally, across pytest invocations => fixture #6033
    1 of 1
    bayandin
  16. python tests: support overlayfs for NeonEnvBuilder.from_repo_dir #6295
  17. client::page_service: fails to parse Error tag #6298
    c/storage/pageserver
    problame
  18. pagebench: fixup after is_rel_block_key changes in #6266 #6303
  19. refactor(NeonEnv): shutdown of child processes #6327
  20. neon_local init: add --force mode that allows an empty dir #6328
  21. pagebench: still depends on pageserver crate #6299
    c/storage/pageserver m/good_first_issue
  22. add script to set up EC2 storage-optimized instance store for benchmarking #6350
  23. fix(pagebench): #6325 broke running without --runtime #6351
  24. pagebench: avoid noise about CopyFail in PS logs #6392
  25. pagebench: use tracing panic hook #6393
  26. fixup(#6037): page_service hangs up within 10ms if there's no message #6388
    jcsp
  27. fixup(#6037): actually fix the issue, #6388 failed to do so #6429
  28. attachment_service: /attach-hook: correctly handle detach #6433
  29. attachment_service: use atomic overwrite to persist attachments.json #6444
  30. fix(pagebench): avoid CopyFail error in success case #6443
  31. fix(attachment_service): corrupted attachments.json when parallel requests #6450
  32. fix(attachment_service): /attach-hook: detachment doesn't survive restart #6457
  33. control_plane/attachment_service: implement PlacementPolicy::Detached #6458
  34. pagebench-based GetPage@LSN performance test #6214
    run-benchmarks
  35. test failures with test_pageserver_max_throughput_getpage_at_latest_lsn[100-13-30] #6473
    4 of 6
  36. test_pageserver_max_throughput_getpage_at_latest_lsn: remove n_tenants=100 combination #6477
  37. run the benchmark nightly on Hetzner runners #6476
    bayandin
  38. run pagebench performance tests on predictable, developer-reproducible infrastructure #6297
    a/ci a/performance c/infra c/storage/pageserver
    Bodobolero

Follow-Ups
Preview Give feedback
  1. TenantInfo: expose generation number #6348
  2. script to duplicate a tenant in neon_local LocalFs remote storage #6349
  3. page_service: ability to exit pagestream mode #6390
    c/storage/pageserver
    problame
@problame problame self-assigned this Nov 2, 2023
@problame problame added c/storage/pageserver Component: storage: pageserver t/Epic Issue type: Epic labels Nov 2, 2023
This was referenced Nov 8, 2023
Closed
Closed
Merged
Closed
problame added a commit that referenced this issue Nov 8, 2023
@problame
cleanup unused RemoteStorage fields (#5830)
e9b227a 
Found this while working on #5771
@problame problame mentioned this issue Nov 22, 2023
Merged
problame added a commit that referenced this issue Nov 22, 2023
@problame
logging: support output to stderr
fadd18a 
(part of the getpage benchmarking epic #5771)

The plan is to make the benchmarking tool log on stderr and emit results
as JSON on stdout. That way, the test suite can simply take captures
stdout and json.loads() it, while interactive users of the benchmarking
tool have a reasonable experience as well.

Existing logging users continue to print to stdout, so, this change
should be a no-op functionally and performance-wise.
problame added a commit that referenced this issue Nov 22, 2023
@problame
logging: support output to stderr (#5896)
9e3c076 
(part of the getpage benchmarking epic #5771)

The plan is to make the benchmarking tool log on stderr and emit results
as JSON on stdout. That way, the test suite can simply take captures
stdout and json.loads() it, while interactive users of the benchmarking
tool have a reasonable experience as well.

Existing logging users continue to print to stdout, so, this change
should be a no-op functionally and performance-wise.
@problame problame mentioned this issue Nov 22, 2023
Closed
problame added a commit that referenced this issue Nov 22, 2023
@problame
logging: support output to stderr
aa87ddb 
(part of the getpage benchmarking epic #5771)

The plan is to make the benchmarking tool log on stderr and emit results
as JSON on stdout. That way, the test suite can simply take captures
stdout and json.loads() it, while interactive users of the benchmarking
tool have a reasonable experience as well.

Existing logging users continue to print to stdout, so, this change
should be a no-op functionally and performance-wise.
This was referenced Nov 27, 2023
Merged
Closed
Merged
problame added a commit that referenced this issue Nov 28, 2023
@problame
test suite: add method for generation-aware detachment of a tenant (#…
286f34d 
…5939)

Part of getpage@lsn benchmark epic:
#5771
problame added a commit that referenced this issue Nov 30, 2023
@problame
pagectl: add subcommand to rewrite layer file summary (#5933)
292281c 
Part of getpage@lsn benchmark epic:
#5771
@problame problame mentioned this issue Dec 4, 2023
Closed
@problame problame mentioned this issue Dec 13, 2023
Draft
problame added a commit that referenced this issue Dec 13, 2023
@problame
add a Rust client for pageserver mgmt api
811cc7e 
Part of getpage@lsn benchmark epic: #5771
This was referenced Dec 13, 2023
Merged
Merged
problame added a commit that referenced this issue Dec 13, 2023
@problame
add a Rust client for Pageserver's page_service
e7449cf 
Part of getpage@lsn benchmark epic: #5771
problame added a commit that referenced this issue Dec 14, 2023
@problame
add a Rust client for pageserver mgmt api
542b074 
Part of getpage@lsn benchmark epic:
#5771

This PR moves the control plane's spread-all-over-the-place client for
the pageserver management API into a separate module within the
pageserver crate.

It also switches to the async version of reqwest, which I think is
generally the right direction, and I need an async client API in the
benchmark epic.
problame added a commit that referenced this issue Dec 14, 2023
@problame
add a Rust client for pageserver mgmt api
4c5b7cf 
Part of getpage@lsn benchmark epic:
#5771

This PR moves the control plane's spread-all-over-the-place client for
the pageserver management API into a separate module within the
pageserver crate.

It also switches to the async version of reqwest, which I think is
generally the right direction, and I need an async client API in the
benchmark epic.
problame added a commit that referenced this issue Dec 14, 2023
@problame
pageserver: keyspace in mgmt api client
2e0737c 
Part of getpage@lsn benchmark epic:
#5771
@problame problame mentioned this issue Dec 14, 2023
Merged
problame added a commit that referenced this issue Dec 15, 2023
@problame
add a Rust client for Pageserver management API (#6127)
1a9854b 
Part of getpage@lsn benchmark epic:
#5771

This PR moves the control plane's spread-all-over-the-place client for
the pageserver management API into a separate module within the
pageserver crate.

I need that client to be async in my benchmarking work, so, this PR
switches to the async version of `reqwest`.
That is also the right direction generally IMO.

The switch to async in turn mandated converting most of the
`control_plane/` code to async.

Note that some of the client methods should be taking `TenantShardId`
instead of `TenantId`, but, none of the callers seem to be
sharding-aware.
Leaving that for another time:
#6154
problame added a commit that referenced this issue Dec 16, 2023
@problame
pageserver: add method to dump keyspace in mgmt api client (#6145)
4787347 
Part of getpage@lsn benchmark epic:
#5771
@problame problame mentioned this issue Dec 18, 2023
Merged
problame added a commit that referenced this issue Dec 18, 2023
@problame
add a Rust client for Pageserver page_service (#6128)
1f9a7d1 
Part of getpage@lsn benchmark epic:
#5771

Stacked atop #6145
@problame problame mentioned this issue Dec 18, 2023
Merged
problame added a commit that referenced this issue Dec 19, 2023
@problame
include timeline ids in tenant details response (#6166)
6ffbbb2 
Part of getpage@lsn benchmark epic:
#5771

This allows getting the list of tenants and timelines without triggering
initial logical size calculation by requesting the timeline details API
response, which would skew our results.
problame added a commit that referenced this issue Dec 21, 2023
@problame
add pageserver component-level benchmark (pagebench) (#6174)
5385791 
This PR adds a component-level benchmarking utility for pageserver.
Its name is `pagebench`.

The problem solved by `pagebench` is that we want to put Pageserver
under high load.

This isn't easily achieved with `pgbench` because it needs to go through
a compute, which has signficant performance overhead compared to
accessing Pageserver directly.

Further, compute has its own performance optimizations (most
importantly: caches). Instead of designing a compute-facing workload
that defeats those internal optimizations, `pagebench` simply bypasses
them by accessing pageserver directly.

Supported benchmarks:

* getpage@latest_lsn
* basebackup
* triggering logical size calculation

This code has no automated users yet.
A performance regression test for getpage@latest_lsn will be added in a
later PR.

part of #5771
@problame problame mentioned this issue Dec 21, 2023
Merged
problame added a commit that referenced this issue Jan 8, 2024
@problame
python tests: support overlayfs for NeonEnvBuilder.from_repo_dir
3d94bd4 
Part of #5771
Extracted from #6214

This PR makes the test suite sensitive to the new env var
`NEON_ENV_BUILDER_FROM_REPO_DIR_USE_OVERLAYFS`.
If it is set, `NeonEnvBuilder.from_repo_dir` uses overlayfs
to duplicate the the snapshot repo dir contents.

Since mounting requires root privileges, we use sudo to perform
the mounts. That, and macOS support, is also why copytree remains
the default.

If we ever run on a filesystem with copy reflink support, we should
consider that as an alternative.

This PR can be tried on a Linux machine on the
`test_backward_compatiblity` test. I took the opportunity to create a
session-scoped fixture for the
compatibility snapshot directory, as a hint to where I hope the
remainder of #6214 will evolve.
@problame problame mentioned this issue Jan 8, 2024
Merged
problame added a commit that referenced this issue Jan 9, 2024
@problame
python tests: support overlayfs for NeonEnvBuilder.from_repo_dir (#6295)
90e0219 
Part of #5771
Extracted from #6214

This PR makes the test suite sensitive to the new env var
`NEON_ENV_BUILDER_FROM_REPO_DIR_USE_OVERLAYFS`.
If it is set, `NeonEnvBuilder.from_repo_dir` uses overlayfs
to duplicate the the snapshot repo dir contents.

Since mounting requires root privileges, we use sudo to perform
the mounts. That, and macOS support, is also why copytree remains
the default.

If we ever run on a filesystem with copy reflink support, we should
consider that as an alternative.

This PR can be tried on a Linux machine on the
`test_backward_compatiblity` test, which uses `from_repo_dir`.
@problame problame mentioned this issue Jan 11, 2024
Merged
problame added a commit that referenced this issue Jan 11, 2024
@problame
neon_local init: add --force mode that allows an empty dir (#6328)
087526b 
Need this in #6214
refs #5771
problame added a commit that referenced this issue Jan 12, 2024
@problame
add script to set up EC2 storage-optimized instance store for benchma…
0b722c1 
…rking

Been using this all the time in #6214

Part of #5771
@problame problame mentioned this issue Jan 12, 2024
Merged
problame added a commit that referenced this issue Jan 12, 2024
@problame
add script to set up EC2 storage-optimized instance store for benchma…
b76454a 
…rking (#6350)

Been using this all the time in
#6214

Part of #5771

Should consider this in #6297
@problame problame mentioned this issue Feb 12, 2024
Open
@jcsp
Copy link
Collaborator

jcsp commented Mar 11, 2024

Status from Alexander:

@problame problame mentioned this issue Apr 30, 2024
Open
@jcsp
Copy link
Collaborator

jcsp commented Sep 23, 2024

@bayandin his one has been stuck as In Progress for a while?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bayandin bayandin

Labels
c/storage/pageserver Component: storage: pageserver t/Epic Issue type: Epic
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bayandin @jcsp @problame