Ability to specify benchmarks that should run synchronously

[peterHakio]

Clear and concise description of the problem

In cases where the code is not only js, but is using a shared resource such as a database benchmarks are difficult to run in isolation.

Example: we are currently testing prisma vs kysely and have written these tests. But since they are using a shared resourse it is not optimal to run them in parrallel.

Here is one of the tests

describe("Prisma vs Kysely 1000 objects", () => {
  bench("Prisma", async () => {
    await prisma.forecast.findMany({
      take: 1000,
    })
  })
  bench("Kysely", async () => {
    await db.selectFrom("Forecast").selectAll().limit(1000).execute()
  })
  bench("Prisma Raw SQL", async () => {
    await prisma.$queryRaw`
      SELECT
        *
      FROM
        "Forecast"
      LIMIT
        1000;
    `
  })

  bench("Kysely Raw SQL", async () => {
    await sql`
      SELECT
        *
      FROM
        "Forecast"
      LIMIT
        1000;
    `.execute(db)
  })
})

Suggested solution

An ability to run vitest bench --synchronous where all promises are awaited and the benchmark tests are performed synchronous.

Alternative

No response

Additional context

No response

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[hi-ogawa]

It looks like multiple describe are running at the same time:

vitest/packages/vitest/src/runtime/runners/benchmark.ts

Lines 38 to 39 in 6eda473

	if (benchmarkSuiteGroup.length)
	await Promise.all(benchmarkSuiteGroup.map(subSuite => runBenchmarkSuite(subSuite, runner)))

However, multiple bench inside single describe are executed sequentially, so your sample code with four bench inside single describe shouldn't interfere each other. Can you check if you have multiple describe?

Though I'm not familiar with the field of benchmarking, but I would feel everything should run sequentially by default and that probably gives better stability. Maybe we can check with what other benchmark framework does in general.

I'm playing with examples like this. Multiple bench files also run in parallel by default, but it can be made sequential by fileParallelism = false:

https://stackblitz.com/edit/vitest-dev-vitest-cdshc4?file=test.log

Sample code

import { bench, describe } from "vitest"
import fs from "node:fs";

const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms))

describe("suite1", () => {
  let i = 0;
  bench("bench1", async () => {
    await fs.promises.appendFile("test.log",`file1, suite1, bench1 = ${i++}\n`)
    await sleep(500);
  }, { time: 0, iterations: 0 })

  let j = 0;
  bench("bench2", async () => {
    await fs.promises.appendFile("test.log",`file1, suite1, bench2 = ${j++}\n`)
    await sleep(500);
  }, { time: 0, iterations: 0 })
})

describe("suite2", () => {
  let i = 0;
  bench("bench1", async () => {
    await fs.promises.appendFile("test.log",`file1, suite2, bench1 = ${i++}\n`)
    await sleep(500);
  }, { time: 0, iterations: 0 })
})

[peterHakio]

Thank you very much. We did what you suggested and it worked fine.

We made implimented it in the vite.config.js like this. Such that we still could have our tests run in parallel

import { defineConfig } from 'vitest/config';

export default defineConfig(({mode})=>({
  test: {
    fileParallelism: mode!=="benchmark",
  },
}));

And regarding if benchmarks should run sequentially. I can see arguments for both. A good first step could be to add a small section for it in the documentation.

[sheremet-va]

Though I'm not familiar with the field of benchmarking, but I would feel everything should run sequentially by default and that probably gives better stability.

I do agree with that. I think we should flatten all benchmarks and use the suite just for reporting.