Skip to content

Commit

Permalink
new ingest benchmark (#9711)
Browse files Browse the repository at this point in the history
## Problem

We have no specific benchmark testing project migration of postgresql
project with existing data into Neon.
Typical steps of such a project migration are
- schema creation in the neon project
- initial COPY of relations
- creation of indexes and constraints
- vacuum analyze

## Summary of changes

Add a periodic benchmark running 9 AM UTC every day.
In each run:
- copy a 200 GiB project that has realistic schema, data, tables,
indexes and constraints from another project into
  - a new Neon project (7 CU fixed)
- an existing tenant, (but new branch and new database) that already has
4 TiB of data
- use pgcopydb tool to automate all steps and parallelize COPY and index
creation
- parse pgcopydb output and report performance metrics in Neon
performance test database

## Logs

This benchmark has been tested first manually and then as part of
benchmarking.yml workflow, example run see

https://github.com/neondatabase/neon/actions/runs/11757679870
  • Loading branch information
Bodobolero authored Nov 11, 2024
1 parent 1aab347 commit 8db84d9
Show file tree
Hide file tree
Showing 2 changed files with 373 additions and 0 deletions.
1 change: 1 addition & 0 deletions .github/actionlint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,4 @@ config-variables:
- REMOTE_STORAGE_AZURE_REGION
- SLACK_UPCOMING_RELEASE_CHANNEL_ID
- DEV_AWS_OIDC_ROLE_ARN
- BENCHMARK_INGEST_TARGET_PROJECTID
Loading

1 comment on commit 8db84d9

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5446 tests run: 5212 passed, 2 failed, 232 skipped (full report)


Failures on Postgres 16

  • test_sharded_ingest[github-actions-selfhosted-1]: release-x86-64
  • test_basebackup_with_high_slru_count[github-actions-selfhosted-10-13-30]: release-x86-64
# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_sharded_ingest[release-pg16-github-actions-selfhosted-1] or test_basebackup_with_high_slru_count[release-pg16-github-actions-selfhosted-10-13-30]"

Code coverage* (full report)

  • functions: 31.7% (7869 of 24806 functions)
  • lines: 49.4% (62267 of 126045 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
8db84d9 at 2024-11-11T20:01:16.228Z :recycle:

Please sign in to comment.