Makefile: fix building wasm-engine when nix is not installed and add task for local size measuring and comparision #4622

miguelff · 2023-12-28T14:07:10Z

This PR:

Fixes build-qe-wasm when nix is not installed on mac 684dffc
Adds a new make task measure-qe-wasm to continuously measure size locally. fab903f

github-actions · 2023-12-28T14:12:14Z

WASM Size

Engine	This PR	Base branch	Diff
WASM	2.338MiB	2.338MiB	0.000B
WASM (gzip)	907.589KiB	907.589KiB	0.000B

github-actions · 2023-12-28T14:18:16Z

🚨 WASM query-engine: 9 benchmark(s) have regressed at least 2%

Full benchmark report

DATABASE_URL="postgresql://postgres:postgres@localhost:5432/bench?schema=imdb_bench&sslmode=disable" \
node --experimental-wasm-modules query-engine/driver-adapters/executor/dist/bench.mjs
cpu: AMD EPYC 7763 64-Core Processor
runtime: node v18.19.0 (x64-linux)

benchmark                   time (avg)             (min … max)       p75       p99      p995
-------------------------------------------------------------- -----------------------------
• movies.findMany() (all - 25000)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline  307.45 ms/iter (305.19 ms … 317.41 ms) 306.71 ms 317.41 ms 317.41 ms
Web Assembly: Latest    314.09 ms/iter (311.49 ms … 319.18 ms)  314.5 ms 319.18 ms 319.18 ms
Web Assembly: Current   401.74 ms/iter (398.29 ms … 408.58 ms)  403.3 ms 408.58 ms 408.58 ms
Node API: Current        236.9 ms/iter (229.18 ms … 247.84 ms) 240.74 ms 247.84 ms 247.84 ms

summary for movies.findMany() (all - 25000)
  Web Assembly: Current
   1.7x slower than Node API: Current
   1.31x slower than Web Assembly: Baseline
   1.28x slower than Web Assembly: Latest

• movies.findMany({ take: 2000 })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline   12.58 ms/iter   (11.82 ms … 18.49 ms)  12.55 ms  18.49 ms  18.49 ms
Web Assembly: Latest     12.86 ms/iter   (11.99 ms … 22.25 ms)  12.56 ms  22.25 ms  22.25 ms
Web Assembly: Current    16.34 ms/iter   (15.71 ms … 24.71 ms)  16.08 ms  24.71 ms  24.71 ms
Node API: Current         9.91 ms/iter    (9.18 ms … 15.24 ms)   9.78 ms  15.24 ms  15.24 ms

summary for movies.findMany({ take: 2000 })
  Web Assembly: Current
   1.65x slower than Node API: Current
   1.3x slower than Web Assembly: Baseline
   1.27x slower than Web Assembly: Latest

• movies.findMany({ where: {...}, take: 2000 })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline    2.01 ms/iter     (1.84 ms … 3.56 ms)   1.98 ms   3.38 ms   3.41 ms
Web Assembly: Latest      2.06 ms/iter     (1.85 ms … 3.59 ms)   1.97 ms   3.54 ms   3.59 ms
Web Assembly: Current     2.59 ms/iter     (2.41 ms … 4.28 ms)   2.53 ms   4.05 ms   4.22 ms
Node API: Current         1.56 ms/iter     (1.49 ms … 1.79 ms)   1.57 ms   1.77 ms   1.78 ms

summary for movies.findMany({ where: {...}, take: 2000 })
  Web Assembly: Current
   1.66x slower than Node API: Current
   1.29x slower than Web Assembly: Baseline
   1.26x slower than Web Assembly: Latest

• movies.findMany({ include: { cast: true } take: 2000 }) (m2m)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline   12.31 ms/iter   (11.88 ms … 15.72 ms)  12.36 ms  15.72 ms  15.72 ms
Web Assembly: Latest     12.44 ms/iter   (11.97 ms … 16.67 ms)   12.4 ms  16.67 ms  16.67 ms
Web Assembly: Current     16.3 ms/iter   (15.74 ms … 21.79 ms)  16.12 ms  21.79 ms  21.79 ms
Node API: Current         9.82 ms/iter     (9.19 ms … 12.8 ms)   9.98 ms   12.8 ms   12.8 ms

summary for movies.findMany({ include: { cast: true } take: 2000 }) (m2m)
  Web Assembly: Current
   1.66x slower than Node API: Current
   1.32x slower than Web Assembly: Baseline
   1.31x slower than Web Assembly: Latest

• movies.findMany({ where: {...}, include: { cast: true } take: 2000 }) (m2m)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline    1.95 ms/iter     (1.84 ms … 2.99 ms)   1.93 ms   2.74 ms   2.76 ms
Web Assembly: Latest      1.94 ms/iter     (1.84 ms … 5.36 ms)   1.92 ms   2.89 ms   3.45 ms
Web Assembly: Current     2.52 ms/iter     (2.41 ms … 4.01 ms)   2.51 ms   3.13 ms   3.53 ms
Node API: Current         1.58 ms/iter        (1.48 ms … 2 ms)    1.6 ms   1.83 ms   1.95 ms

summary for movies.findMany({ where: {...}, include: { cast: true } take: 2000 }) (m2m)
  Web Assembly: Current
   1.6x slower than Node API: Current
   1.3x slower than Web Assembly: Latest
   1.29x slower than Web Assembly: Baseline

• movies.findMany({ take: 2000, include: { cast: { include: { person: true } } } })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline   12.37 ms/iter   (11.92 ms … 16.72 ms)  12.43 ms  16.72 ms  16.72 ms
Web Assembly: Latest     12.34 ms/iter   (11.97 ms … 14.71 ms)  12.41 ms  14.71 ms  14.71 ms
Web Assembly: Current    16.17 ms/iter    (15.7 ms … 18.94 ms)  16.15 ms  18.94 ms  18.94 ms
Node API: Current         9.52 ms/iter     (9.1 ms … 11.48 ms)   9.68 ms  11.48 ms  11.48 ms

summary for movies.findMany({ take: 2000, include: { cast: { include: { person: true } } } })
  Web Assembly: Current
   1.7x slower than Node API: Current
   1.31x slower than Web Assembly: Latest
   1.31x slower than Web Assembly: Baseline

• movie.findMany({ where: { ... }, take: 2000, include: { cast: { include: { person: true } } } })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline     1.9 ms/iter     (1.82 ms … 2.78 ms)    1.9 ms   2.38 ms   2.39 ms
Web Assembly: Latest       1.9 ms/iter     (1.83 ms … 2.89 ms)    1.9 ms   2.51 ms   2.56 ms
Web Assembly: Current     2.46 ms/iter      (2.4 ms … 2.86 ms)   2.47 ms   2.68 ms    2.7 ms
Node API: Current         1.59 ms/iter     (1.51 ms … 1.83 ms)    1.6 ms    1.8 ms   1.82 ms

summary for movie.findMany({ where: { ... }, take: 2000, include: { cast: { include: { person: true } } } })
  Web Assembly: Current
   1.55x slower than Node API: Current
   1.3x slower than Web Assembly: Baseline
   1.29x slower than Web Assembly: Latest

• movie.findMany({ where: { reviews: { author: { ... } }, take: 100 }) (to-many -> to-one)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline  951.42 µs/iter   (890.03 µs … 1.53 ms) 944.51 µs   1.38 ms   1.43 ms
Web Assembly: Latest    941.03 µs/iter   (881.33 µs … 1.47 ms) 933.95 µs   1.43 ms   1.45 ms
Web Assembly: Current     1.26 ms/iter     (1.17 ms … 2.21 ms)   1.25 ms   1.97 ms   1.99 ms
Node API: Current       853.64 µs/iter   (755.76 µs … 1.47 ms) 886.28 µs   1.24 ms   1.28 ms

summary for movie.findMany({ where: { reviews: { author: { ... } }, take: 100 }) (to-many -> to-one)
  Web Assembly: Current
   1.48x slower than Node API: Current
   1.34x slower than Web Assembly: Latest
   1.33x slower than Web Assembly: Baseline

• movie.findMany({ where: { cast: { person: { ... } }, take: 100 }) (m2m -> to-one)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline  923.04 µs/iter   (866.37 µs … 3.38 ms) 924.75 µs   1.29 ms   1.43 ms
Web Assembly: Latest    927.57 µs/iter   (866.87 µs … 1.43 ms) 930.16 µs   1.28 ms   1.32 ms
Web Assembly: Current     1.23 ms/iter     (1.15 ms … 3.71 ms)   1.22 ms   1.72 ms   1.78 ms
Node API: Current       858.89 µs/iter   (785.94 µs … 1.11 ms) 869.45 µs 999.96 µs   1.02 ms

summary for movie.findMany({ where: { cast: { person: { ... } }, take: 100 }) (m2m -> to-one)
  Web Assembly: Current
   1.44x slower than Node API: Current
   1.34x slower than Web Assembly: Baseline
   1.33x slower than Web Assembly: Latest

After changes in db1f45c

codspeed-hq · 2023-12-28T14:19:14Z

CodSpeed Performance Report

Merging #4622 will not alter performance

_{Comparing wasm-make-improvements (db1f45c) with main (b157f36)}

Summary

✅ 11 untouched benchmarks

jkomyno · 2023-12-29T11:48:54Z

Makefile

@@ -327,14 +327,26 @@ build-qe-napi:
 	cargo build --package query-engine-node-api --profile $(PROFILE)

 build-qe-wasm:
-ifndef $(NIX)
+ifneq ($(strip $(NIX)),)


Rewier's note: "The strip function is used to remove leading and trailing whitespace from the value of the variable". I'm not quite sure why this is needed, but it's safe to keep.

this is needed because depending on your shell command -v nix (Which is used to populate the NIX variable) might not return what you want for the comparision to be effective.

jkomyno · 2023-12-29T11:51:01Z

Makefile

+	echo "Previous size: `cat size.txt`"; \
+	echo "Current size: `cat temp_size.txt`"; \


Non-blocking nit: can we call this .size_previous.txt and .size_current.txt?
I feel the current names for these text files could be a source of confusion.

Also, let's note that this only measures the gzip'd wasm size without considering the necessary JavaScript glue code. On the other hand, wasm-pack pack creates a tar.gz file that takes the glue code into account as well.

How much code does that account for, zipped?

miguelff requested a review from a team as a code owner December 28, 2023 14:07

miguelff requested review from laplab and jkomyno and removed request for a team December 28, 2023 14:07

jkomyno reviewed Dec 29, 2023

View reviewed changes

apolanc assigned miguelff Jan 11, 2024

miguelff force-pushed the wasm-make-improvements branch 2 times, most recently from ee5a2b2 to 8c38ba4 Compare January 16, 2024 17:33

jkomyno approved these changes Jan 17, 2024

View reviewed changes

Fix build-qe-wasm when nix is not installed

21ffa38

miguelff force-pushed the wasm-make-improvements branch from 8c38ba4 to 311e20b Compare January 17, 2024 15:17

Add measure-qe-wasm task to continuously measure size locally

db1f45c

miguelff force-pushed the wasm-make-improvements branch from 311e20b to db1f45c Compare January 17, 2024 15:39

miguelff merged commit 7f4fc14 into main Jan 17, 2024
123 of 126 checks passed

miguelff deleted the wasm-make-improvements branch January 17, 2024 15:41

Jolg42 added this to the 5.9.0 milestone Jan 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Makefile: fix building wasm-engine when nix is not installed and add task for local size measuring and comparision #4622

Makefile: fix building wasm-engine when nix is not installed and add task for local size measuring and comparision #4622

miguelff commented Dec 28, 2023 •

edited

Loading

github-actions bot commented Dec 28, 2023 •

edited

Loading

github-actions bot commented Dec 28, 2023 •

edited

Loading

codspeed-hq bot commented Dec 28, 2023 •

edited

Loading

jkomyno Dec 29, 2023

miguelff Jan 16, 2024

jkomyno Dec 29, 2023

jkomyno Jan 8, 2024

miguelff Jan 16, 2024 •

edited

Loading

		echo "Previous size: `cat size.txt`"; \
		echo "Current size: `cat temp_size.txt`"; \

Makefile: fix building wasm-engine when nix is not installed and add task for local size measuring and comparision #4622

Makefile: fix building wasm-engine when nix is not installed and add task for local size measuring and comparision #4622

Conversation

miguelff commented Dec 28, 2023 • edited Loading

github-actions bot commented Dec 28, 2023 • edited Loading

WASM Size

github-actions bot commented Dec 28, 2023 • edited Loading

🚨 WASM query-engine: 9 benchmark(s) have regressed at least 2%

codspeed-hq bot commented Dec 28, 2023 • edited Loading

CodSpeed Performance Report

Merging #4622 will not alter performance

Summary

jkomyno Dec 29, 2023

Choose a reason for hiding this comment

miguelff Jan 16, 2024

Choose a reason for hiding this comment

jkomyno Dec 29, 2023

Choose a reason for hiding this comment

jkomyno Jan 8, 2024

Choose a reason for hiding this comment

miguelff Jan 16, 2024 • edited Loading

Choose a reason for hiding this comment

miguelff commented Dec 28, 2023 •

edited

Loading

github-actions bot commented Dec 28, 2023 •

edited

Loading

github-actions bot commented Dec 28, 2023 •

edited

Loading

codspeed-hq bot commented Dec 28, 2023 •

edited

Loading

miguelff Jan 16, 2024 •

edited

Loading