Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ingest benchmark: after effective_io_concurrency = 100 we can increase compute side parallelism #9904

Merged
merged 4 commits into from
Nov 27, 2024

Conversation

Bodobolero
Copy link
Contributor

Problem

ingest benchmark tests project migration to Neon involving steps

  • COPY relation data
  • create indexes
  • create constraints

Previously we used only 4 copy jobs, 4 create index jobs and 7 maintenance workers. After increasing effective_io_concurrency on compute we see that we can sustain more parallelism in the ingest bench

Summary of changes

Increase copy jobs to 8, create index jobs to 8 and maintenance workers to 16

@Bodobolero Bodobolero requested a review from a team November 27, 2024 09:08
Copy link
Contributor

@fedordikarev fedordikarev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@Bodobolero Bodobolero enabled auto-merge November 27, 2024 09:23
Copy link
Member

@bayandin bayandin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@Bodobolero Bodobolero added this pull request to the merge queue Nov 27, 2024
Merged via the queue into main with commit 277c33b Nov 27, 2024
78 checks passed
@Bodobolero Bodobolero deleted the bodobolero/test_ingest_parallelism branch November 27, 2024 10:11
Copy link

6941 tests run: 6633 passed, 0 failed, 308 skipped (full report)


Flaky tests (3)

Postgres 17

Postgres 15

Postgres 14

Code coverage* (full report)

  • functions: 30.8% (7975 of 25851 functions)
  • lines: 48.6% (63303 of 130182 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
44ac7c8 at 2024-11-27T10:12:29.686Z :recycle:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants