new ingest benchmark

[Bodobolero]

Problem

We have no specific benchmark testing project migration of postgresql project with existing data into Neon.
Typical steps of such a project migration are

• schema creation in the neon project
• initial COPY of relations
• creation of indexes and constraints
• vacuum analyze

Summary of changes

Add a periodic benchmark running 9 AM UTC every day.
In each run:

• copy a 200 GiB project that has realistic schema, data, tables, indexes and constraints from another project into
  • a new Neon project (7 CU fixed)
  • an existing tenant, (but new branch and new database) that already has 4 TiB of data
• use pgcopydb tool to automate all steps and parallelize COPY and index creation
• parse pgcopydb output and report performance metrics in Neon performance test database

Logs

This benchmark has been tested first manually and then as part of benchmarking.yml workflow, example run see

https://github.com/neondatabase/neon/actions/runs/11757679870

[github-actions]

5355 tests run: 5133 passed, 0 failed, 222 skipped (full report)

---

Flaky tests (3)

Postgres 17

• test_create_churn_during_restart: debug-x86-64

Postgres 16

• test_isolation[None]: release-arm64

Postgres 15

• test_replica_query_race: release-x86-64

Code coverage* (full report)

• functions: 31.7% (7869 of 24806 functions)
• lines: 49.4% (62259 of 126036 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
69bce45 at 2024-11-11T14:08:04.161Z :recycle:}

[bayandin]

I'm okay with merging it, but I think it's best to wrap it in a Python test.
(we can merge the PR, start getting results and convert it to a Python test in parallel)