Fix benchmark reporting when benchmark script fails, and provide more reliable and informative results #4623

miguelff · 2023-12-28T15:56:45Z

Close https://github.com/prisma/team-orm/issues/851

We found a bug reporting the benchmarks when the benchmarks errored. This PR fixes that and also improves overall benchmark reporting.

The script has been tested with an artificial benchmark results file to:

Report benchmark improvements
Report benchmark regressions
Report no substantial changes

And to take into consideration a failed benchmarking script.

github-actions · 2024-01-22T17:51:06Z

WASM Size

Engine	This PR	Base branch	Diff
WASM	2.334MiB	2.334MiB	0.000B
WASM (gzip)	907.092KiB	907.092KiB	0.000B

codspeed-hq · 2024-01-22T17:57:23Z

CodSpeed Performance Report

Merging #4623 will not alter performance

_{Comparing improve-benchmark-reporting (7cb15b7) with main (e5078ed)}

Summary

✅ 11 untouched benchmarks

github-actions · 2024-01-23T05:50:01Z

✅ WASM query-engine performance won't change substantially (0.992x)

Full benchmark report

DATABASE_URL="postgresql://postgres:postgres@localhost:5432/bench?schema=imdb_bench&sslmode=disable" \
node --experimental-wasm-modules query-engine/driver-adapters/executor/dist/bench.mjs
cpu: AMD EPYC 7763 64-Core Processor
runtime: node v18.19.0 (x64-linux)

benchmark                   time (avg)             (min … max)       p75       p99      p995
-------------------------------------------------------------- -----------------------------
• movies.findMany() (all - 25000)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline  310.21 ms/iter (307.71 ms … 317.13 ms) 310.81 ms 317.13 ms 317.13 ms
Web Assembly: Latest    405.05 ms/iter  (401.8 ms … 410.78 ms)  406.7 ms 410.78 ms 410.78 ms
Web Assembly: Current   404.08 ms/iter (400.58 ms … 416.11 ms)  403.7 ms 416.11 ms 416.11 ms
Node API: Current        244.5 ms/iter (237.88 ms … 252.24 ms) 246.75 ms 252.24 ms 252.24 ms

summary for movies.findMany() (all - 25000)
  Web Assembly: Current
   1.65x slower than Node API: Current
   1.3x slower than Web Assembly: Baseline
   1x faster than Web Assembly: Latest

• movies.findMany({ take: 2000 })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline   12.67 ms/iter   (11.84 ms … 17.52 ms)  12.67 ms  17.52 ms  17.52 ms
Web Assembly: Latest     16.33 ms/iter   (15.88 ms … 22.77 ms)  16.15 ms  22.77 ms  22.77 ms
Web Assembly: Current    16.14 ms/iter    (15.7 ms … 17.71 ms)  16.22 ms  17.71 ms  17.71 ms
Node API: Current         9.31 ms/iter     (9.13 ms … 9.81 ms)   9.36 ms   9.81 ms   9.81 ms

summary for movies.findMany({ take: 2000 })
  Web Assembly: Current
   1.73x slower than Node API: Current
   1.27x slower than Web Assembly: Baseline
   1.01x faster than Web Assembly: Latest

• movies.findMany({ where: {...}, take: 2000 })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline    2.02 ms/iter     (1.86 ms … 3.43 ms)   1.96 ms   3.39 ms   3.41 ms
Web Assembly: Latest      2.62 ms/iter      (2.45 ms … 4.6 ms)   2.59 ms    4.3 ms   4.34 ms
Web Assembly: Current     2.59 ms/iter     (2.46 ms … 4.54 ms)   2.55 ms   4.24 ms   4.25 ms
Node API: Current         1.55 ms/iter     (1.46 ms … 2.75 ms)   1.54 ms   2.26 ms   2.36 ms

summary for movies.findMany({ where: {...}, take: 2000 })
  Web Assembly: Current
   1.67x slower than Node API: Current
   1.28x slower than Web Assembly: Baseline
   1.01x faster than Web Assembly: Latest

• movies.findMany({ include: { cast: true } take: 2000 }) (m2m)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline   12.24 ms/iter   (11.85 ms … 13.72 ms)  12.35 ms  13.72 ms  13.72 ms
Web Assembly: Latest     16.49 ms/iter   (16.14 ms … 21.47 ms)  16.42 ms  21.47 ms  21.47 ms
Web Assembly: Current    16.06 ms/iter   (15.59 ms … 20.91 ms)  16.09 ms  20.91 ms  20.91 ms
Node API: Current         9.41 ms/iter     (9.1 ms … 14.61 ms)   9.41 ms  14.61 ms  14.61 ms

summary for movies.findMany({ include: { cast: true } take: 2000 }) (m2m)
  Web Assembly: Current
   1.71x slower than Node API: Current
   1.31x slower than Web Assembly: Baseline
   1.03x faster than Web Assembly: Latest

• movies.findMany({ where: {...}, include: { cast: true } take: 2000 }) (m2m)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline    1.94 ms/iter     (1.85 ms … 4.88 ms)   1.93 ms   2.63 ms   2.91 ms
Web Assembly: Latest      2.52 ms/iter     (2.44 ms … 3.29 ms)   2.52 ms   3.04 ms    3.1 ms
Web Assembly: Current     2.58 ms/iter      (2.44 ms … 4.3 ms)   2.53 ms   3.83 ms   4.29 ms
Node API: Current         1.54 ms/iter     (1.46 ms … 2.64 ms)   1.55 ms   1.83 ms   2.25 ms

summary for movies.findMany({ where: {...}, include: { cast: true } take: 2000 }) (m2m)
  Web Assembly: Current
   1.68x slower than Node API: Current
   1.33x slower than Web Assembly: Baseline
   1.02x slower than Web Assembly: Latest

• movies.findMany({ take: 2000, include: { cast: { include: { person: true } } } })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline    12.3 ms/iter   (11.87 ms … 16.97 ms)   12.3 ms  16.97 ms  16.97 ms
Web Assembly: Latest     16.33 ms/iter      (16 ms … 18.05 ms)  16.35 ms  18.05 ms  18.05 ms
Web Assembly: Current       16 ms/iter   (15.64 ms … 16.54 ms)  16.13 ms  16.54 ms  16.54 ms
Node API: Current         9.39 ms/iter      (8.96 ms … 9.9 ms)   9.44 ms    9.9 ms    9.9 ms

summary for movies.findMany({ take: 2000, include: { cast: { include: { person: true } } } })
  Web Assembly: Current
   1.7x slower than Node API: Current
   1.3x slower than Web Assembly: Baseline
   1.02x faster than Web Assembly: Latest

• movie.findMany({ where: { ... }, take: 2000, include: { cast: { include: { person: true } } } })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline    1.91 ms/iter     (1.82 ms … 2.96 ms)    1.9 ms    2.8 ms   2.85 ms
Web Assembly: Latest      2.51 ms/iter      (2.42 ms … 3.9 ms)    2.5 ms   3.18 ms   3.31 ms
Web Assembly: Current      2.5 ms/iter     (2.41 ms … 3.92 ms)   2.49 ms   3.73 ms   3.88 ms
Node API: Current         1.55 ms/iter      (1.45 ms … 1.8 ms)   1.57 ms   1.78 ms   1.79 ms

summary for movie.findMany({ where: { ... }, take: 2000, include: { cast: { include: { person: true } } } })
  Web Assembly: Current
   1.61x slower than Node API: Current
   1.31x slower than Web Assembly: Baseline
   1x faster than Web Assembly: Latest

• movie.findMany({ where: { reviews: { author: { ... } }, take: 100 }) (to-many -> to-one)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline  916.54 µs/iter   (858.02 µs … 1.46 ms)  913.9 µs   1.29 ms   1.42 ms
Web Assembly: Latest      1.22 ms/iter     (1.15 ms … 2.04 ms)   1.21 ms   1.87 ms   1.96 ms
Web Assembly: Current     1.22 ms/iter     (1.16 ms … 2.03 ms)   1.21 ms   1.59 ms   1.73 ms
Node API: Current       844.36 µs/iter    (770.8 µs … 1.06 ms) 855.73 µs 957.36 µs  990.7 µs

summary for movie.findMany({ where: { reviews: { author: { ... } }, take: 100 }) (to-many -> to-one)
  Web Assembly: Current
   1.44x slower than Node API: Current
   1.33x slower than Web Assembly: Baseline
   1x faster than Web Assembly: Latest

• movie.findMany({ where: { cast: { person: { ... } }, take: 100 }) (m2m -> to-one)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline   912.8 µs/iter    (858.52 µs … 1.6 ms) 911.03 µs   1.28 ms   1.58 ms
Web Assembly: Latest      1.21 ms/iter     (1.15 ms … 1.87 ms)   1.22 ms   1.54 ms    1.8 ms
Web Assembly: Current      1.2 ms/iter     (1.15 ms … 1.56 ms)    1.2 ms   1.45 ms    1.5 ms
Node API: Current       853.38 µs/iter   (761.47 µs … 1.65 ms) 862.03 µs   1.24 ms   1.28 ms

summary for movie.findMany({ where: { cast: { person: { ... } }, take: 100 }) (m2m -> to-one)
  Web Assembly: Current
   1.4x slower than Node API: Current
   1.31x slower than Web Assembly: Baseline
   1.02x faster than Web Assembly: Latest

After changes in 7cb15b7

Jolg42 · 2024-01-23T08:58:08Z

Logs from the run here:
https://github.com/prisma/prisma-engines/actions/runs/7621926473/job/20759106287?pr=4623#step:9:1

And in the comment up
#4623 (comment)

Looks good to me ✨ (only had a quick look)

jkomyno · 2024-01-23T09:15:59Z

query-engine/driver-adapters/executor/src/recording.ts

+  // Recording is currently only used in benchmarks. Before we used to serialize the whole query
+  // (sql + args) but since bigints are not serialized by JSON.stringify, and we didn't really need
+  // args for benchmarks, we just serialize the sql part.
+  //
+  // If this ever changes (we reuse query recording in tests) we need to make sure to serialize the
+  // args as well.
+  const queryToKey = (query) => JSON.stringify(query.sql);


Reviewer's note: as discussed with Miguel yesterday, this PR also exists: #4667

miguelff requested a review from a team as a code owner December 28, 2023 15:56

miguelff requested review from Weakky and Druue and removed request for a team December 28, 2023 15:56

miguelff force-pushed the improve-benchmark-reporting branch from be75443 to 726141e Compare December 28, 2023 16:30

miguelff marked this pull request as draft December 28, 2023 17:12

miguelff force-pushed the wasm-make-improvements branch 4 times, most recently from 311e20b to db1f45c Compare January 17, 2024 15:39

Base automatically changed from wasm-make-improvements to main January 17, 2024 15:41

Improve benchmark reporting

3626014

miguelff force-pushed the improve-benchmark-reporting branch from 05780f9 to d572b01 Compare January 22, 2024 15:41

miguelff added 4 commits January 22, 2024 18:32

Serialize only sql in query recording for the benchmarks

c33d75b

Exit when script doesn't finish properly

57f6d98

Fix benchmark workflow

e520dff

No need to output when something went wrong

81cf26f

miguelff force-pushed the improve-benchmark-reporting branch from aa278ce to 2ecda16 Compare January 22, 2024 17:46

Install bc

c3fc98c

miguelff force-pushed the improve-benchmark-reporting branch from 2ecda16 to c3fc98c Compare January 22, 2024 18:13

miguelff changed the title ~~Improve benchmark reporting~~ Fix benchmark reporting when benchmark script fails Jan 22, 2024

miguelff self-assigned this Jan 22, 2024

miguelff force-pushed the improve-benchmark-reporting branch 6 times, most recently from b02a28e to bd1b405 Compare January 23, 2024 05:39

miguelff force-pushed the improve-benchmark-reporting branch 6 times, most recently from 5140d1e to 7ee4e62 Compare January 23, 2024 06:16

Fix workflow after debugging and QA

cb86c39

miguelff force-pushed the improve-benchmark-reporting branch from 7ee4e62 to cb86c39 Compare January 23, 2024 06:20

miguelff changed the title ~~Fix benchmark reporting when benchmark script fails~~ Fix benchmark reporting when benchmark script fails, and provide more reliable and informative results Jan 23, 2024

miguelff force-pushed the improve-benchmark-reporting branch from bcfd5fc to 0105677 Compare January 23, 2024 06:29

Small change to report of non substantial changes

7cb15b7

miguelff force-pushed the improve-benchmark-reporting branch from 0105677 to 7cb15b7 Compare January 23, 2024 06:30

miguelff marked this pull request as ready for review January 23, 2024 06:30

miguelff added this to the 5.9.0 milestone Jan 23, 2024

jkomyno reviewed Jan 23, 2024

View reviewed changes

jkomyno approved these changes Jan 23, 2024

View reviewed changes

miguelff merged commit 6d02932 into main Jan 23, 2024
112 checks passed

miguelff deleted the improve-benchmark-reporting branch January 23, 2024 09:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix benchmark reporting when benchmark script fails, and provide more reliable and informative results #4623

Fix benchmark reporting when benchmark script fails, and provide more reliable and informative results #4623

miguelff commented Dec 28, 2023 •

edited

Loading

github-actions bot commented Jan 22, 2024 •

edited

Loading

codspeed-hq bot commented Jan 22, 2024 •

edited

Loading

github-actions bot commented Jan 23, 2024 •

edited

Loading

Jolg42 commented Jan 23, 2024 •

edited

Loading

jkomyno Jan 23, 2024

Fix benchmark reporting when benchmark script fails, and provide more reliable and informative results #4623

Fix benchmark reporting when benchmark script fails, and provide more reliable and informative results #4623

Conversation

miguelff commented Dec 28, 2023 • edited Loading

github-actions bot commented Jan 22, 2024 • edited Loading

WASM Size

codspeed-hq bot commented Jan 22, 2024 • edited Loading

CodSpeed Performance Report

Merging #4623 will not alter performance

Summary

github-actions bot commented Jan 23, 2024 • edited Loading

✅ WASM query-engine performance won't change substantially (0.992x)

Jolg42 commented Jan 23, 2024 • edited Loading

jkomyno Jan 23, 2024

Choose a reason for hiding this comment

miguelff commented Dec 28, 2023 •

edited

Loading

github-actions bot commented Jan 22, 2024 •

edited

Loading

codspeed-hq bot commented Jan 22, 2024 •

edited

Loading

github-actions bot commented Jan 23, 2024 •

edited

Loading

Jolg42 commented Jan 23, 2024 •

edited

Loading