Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[experiment] Run babel over output for let/const to var #52656

Closed
wants to merge 17 commits into from

Conversation

jakebailey
Copy link
Member

@jakebailey jakebailey commented Feb 7, 2023

This is a performance boost we already knew we'd be losing when switching to modules + esbuild, as the latter doesn't support this downleveling (yet?).

I want to retest to see how good this would be, it's probably going to be worth it to do some form of this.

@jakebailey
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Feb 7, 2023

Heya @jakebailey, I've started to run the perf test suite on this PR at 1a42515. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - main..52656
Metric main 52656 Delta Best Worst p-value
Angular - node (v18.10.0, x64)
Memory used 359,797k (± 0.02%) 358,447k (± 0.03%) -1,350k (- 0.38%) 358,300k 358,548k p=0.005 n=6
Parse Time 3.93s (± 0.90%) 3.52s (± 0.45%) 🟩-0.41s (-10.44%) 3.50s 3.54s p=0.005 n=6
Bind Time 1.17s (± 0.64%) 1.13s (± 0.46%) 🟩-0.04s (- 3.27%) 1.13s 1.14s p=0.004 n=6
Check Time 8.71s (± 0.72%) 8.46s (± 0.43%) -0.25s (- 2.93%) 8.42s 8.52s p=0.005 n=6
Emit Time 7.61s (± 0.87%) 7.39s (± 1.32%) -0.22s (- 2.91%) 7.26s 7.55s p=0.008 n=6
Total Time 21.42s (± 0.68%) 20.50s (± 0.67%) 🟩-0.92s (- 4.29%) 20.34s 20.73s p=0.005 n=6
Compiler-Unions - node (v18.10.0, x64)
Memory used 193,220k (± 1.19%) 194,214k (± 1.58%) ~ 191,380k 197,145k p=0.689 n=6
Parse Time 1.63s (± 1.32%) 1.48s (± 0.35%) 🟩-0.15s (- 9.28%) 1.48s 1.49s p=0.004 n=6
Bind Time 0.80s (± 0.65%) 0.77s (± 0.67%) 🟩-0.03s (- 3.77%) 0.76s 0.77s p=0.004 n=6
Check Time 9.75s (± 0.52%) 9.56s (± 0.41%) -0.19s (- 1.97%) 9.51s 9.61s p=0.005 n=6
Emit Time 2.80s (± 1.01%) 2.73s (± 1.05%) -0.07s (- 2.38%) 2.69s 2.77s p=0.008 n=6
Total Time 14.98s (± 0.46%) 14.54s (± 0.40%) -0.44s (- 2.90%) 14.48s 14.61s p=0.005 n=6
Monaco - node (v18.10.0, x64)
Memory used 343,965k (± 0.01%) 342,807k (± 0.01%) -1,159k (- 0.34%) 342,750k 342,834k p=0.005 n=6
Parse Time 2.94s (± 0.84%) 2.60s (± 1.13%) 🟩-0.33s (-11.40%) 2.55s 2.63s p=0.005 n=6
Bind Time 1.04s (± 1.32%) 1.02s (± 1.01%) ~ 1.01s 1.04s p=0.098 n=6
Check Time 7.07s (± 0.47%) 6.89s (± 0.35%) -0.18s (- 2.57%) 6.86s 6.91s p=0.005 n=6
Emit Time 4.33s (± 1.22%) 4.21s (± 0.32%) -0.12s (- 2.73%) 4.19s 4.23s p=0.005 n=6
Total Time 15.37s (± 0.45%) 14.72s (± 0.33%) 🟩-0.64s (- 4.19%) 14.68s 14.79s p=0.005 n=6
TFS - node (v18.10.0, x64)
Memory used 300,377k (± 0.01%) 299,321k (± 0.01%) -1,055k (- 0.35%) 299,297k 299,343k p=0.005 n=6
Parse Time 2.27s (± 1.71%) 2.10s (± 1.11%) 🟩-0.18s (- 7.77%) 2.07s 2.13s p=0.005 n=6
Bind Time 1.17s (± 0.44%) 1.14s (± 0.45%) -0.03s (- 2.57%) 1.13s 1.14s p=0.004 n=6
Check Time 6.54s (± 0.65%) 6.47s (± 0.62%) -0.08s (- 1.20%) 6.41s 6.51s p=0.020 n=6
Emit Time 3.93s (± 1.01%) 3.82s (± 0.66%) -0.11s (- 2.76%) 3.79s 3.85s p=0.005 n=6
Total Time 13.92s (± 0.51%) 13.52s (± 0.35%) -0.40s (- 2.85%) 13.45s 13.57s p=0.005 n=6
material-ui - node (v18.10.0, x64)
Memory used 476,782k (± 0.01%) 475,737k (± 0.01%) -1,045k (- 0.22%) 475,696k 475,792k p=0.005 n=6
Parse Time 3.53s (± 0.46%) 3.11s (± 0.24%) 🟩-0.42s (-11.77%) 3.10s 3.12s p=0.005 n=6
Bind Time 0.97s (± 0.53%) 0.93s (± 0.87%) 🟩-0.03s (- 3.45%) 0.92s 0.94s p=0.004 n=6
Check Time 17.12s (± 0.55%) 17.06s (± 0.63%) ~ 16.89s 17.21s p=0.469 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 21.61s (± 0.38%) 21.11s (± 0.52%) -0.51s (- 2.35%) 20.93s 21.25s p=0.005 n=6
xstate - node (v18.10.0, x64)
Memory used 549,092k (± 0.03%) 546,665k (± 0.02%) -2,426k (- 0.44%) 546,562k 546,807k p=0.005 n=6
Parse Time 4.58s (± 0.67%) 4.03s (± 0.36%) 🟩-0.55s (-12.07%) 4.01s 4.05s p=0.005 n=6
Bind Time 1.71s (± 1.26%) 1.67s (± 0.82%) -0.05s (- 2.82%) 1.65s 1.69s p=0.006 n=6
Check Time 2.81s (± 0.54%) 2.75s (± 0.92%) -0.06s (- 2.31%) 2.73s 2.80s p=0.007 n=6
Emit Time 0.08s (± 0.00%) 0.08s (± 6.19%) ~ 0.08s 0.09s p=0.174 n=6
Total Time 9.20s (± 0.25%) 8.53s (± 0.38%) 🟩-0.67s (- 7.32%) 8.49s 8.59s p=0.005 n=6
Angular - node (v16.17.1, x64)
Memory used 359,144k (± 0.01%) 357,763k (± 0.01%) -1,381k (- 0.38%) 357,733k 357,806k p=0.005 n=6
Parse Time 4.17s (± 0.45%) 3.69s (± 0.44%) 🟩-0.47s (-11.32%) 3.68s 3.72s p=0.005 n=6
Bind Time 1.24s (± 0.41%) 1.19s (± 0.86%) 🟩-0.05s (- 4.02%) 1.18s 1.21s p=0.004 n=6
Check Time 9.49s (± 0.19%) 9.23s (± 0.23%) -0.26s (- 2.72%) 9.21s 9.27s p=0.005 n=6
Emit Time 8.06s (± 0.35%) 7.91s (± 0.73%) -0.15s (- 1.90%) 7.84s 7.99s p=0.004 n=6
Total Time 22.96s (± 0.23%) 22.02s (± 0.32%) 🟩-0.94s (- 4.07%) 21.96s 22.14s p=0.005 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 193,814k (± 0.04%) 193,031k (± 0.02%) -783k (- 0.40%) 192,959k 193,064k p=0.005 n=6
Parse Time 1.80s (± 0.93%) 1.57s (± 0.87%) 🟩-0.23s (-12.96%) 1.55s 1.59s p=0.005 n=6
Bind Time 0.84s (± 0.00%) 0.82s (± 0.50%) -0.02s (- 2.18%) 0.82s 0.83s p=0.002 n=6
Check Time 10.42s (± 0.35%) 10.10s (± 0.73%) 🟩-0.32s (- 3.04%) 10.01s 10.20s p=0.005 n=6
Emit Time 3.10s (± 4.70%) 3.01s (± 0.91%) 🟩-0.10s (- 3.17%) 2.97s 3.05s p=0.024 n=6
Total Time 16.17s (± 1.07%) 15.50s (± 0.38%) 🟩-0.67s (- 4.16%) 15.41s 15.55s p=0.005 n=6
Monaco - node (v16.17.1, x64)
Memory used 343,232k (± 0.01%) 342,073k (± 0.01%) -1,159k (- 0.34%) 342,035k 342,106k p=0.005 n=6
Parse Time 3.14s (± 1.50%) 2.78s (± 1.13%) 🟩-0.36s (-11.52%) 2.74s 2.82s p=0.005 n=6
Bind Time 1.11s (± 1.51%) 1.08s (± 0.48%) -0.03s (- 2.40%) 1.08s 1.09s p=0.024 n=6
Check Time 7.73s (± 0.25%) 7.50s (± 0.39%) 🟩-0.23s (- 3.02%) 7.46s 7.53s p=0.005 n=6
Emit Time 4.54s (± 0.46%) 4.46s (± 0.94%) -0.08s (- 1.73%) 4.40s 4.51s p=0.005 n=6
Total Time 16.51s (± 0.50%) 15.82s (± 0.50%) 🟩-0.70s (- 4.22%) 15.69s 15.90s p=0.005 n=6
TFS - node (v16.17.1, x64)
Memory used 299,706k (± 0.01%) 298,645k (± 0.01%) -1,061k (- 0.35%) 298,626k 298,663k p=0.005 n=6
Parse Time 2.46s (± 0.99%) 2.21s (± 0.50%) 🟩-0.25s (-10.16%) 2.20s 2.22s p=0.004 n=6
Bind Time 1.26s (± 0.82%) 1.23s (± 1.03%) -0.03s (- 2.12%) 1.22s 1.25s p=0.012 n=6
Check Time 7.21s (± 0.39%) 7.09s (± 0.61%) -0.12s (- 1.66%) 7.05s 7.15s p=0.005 n=6
Emit Time 4.22s (± 0.21%) 4.31s (± 0.76%) +0.09s (+ 2.05%) 4.27s 4.35s p=0.005 n=6
Total Time 15.15s (± 0.23%) 14.84s (± 0.46%) -0.31s (- 2.04%) 14.76s 14.94s p=0.005 n=6
material-ui - node (v16.17.1, x64)
Memory used 476,012k (± 0.01%) 475,060k (± 0.01%) -952k (- 0.20%) 475,033k 475,101k p=0.005 n=6
Parse Time 3.68s (± 0.14%) 3.29s (± 0.23%) 🟩-0.39s (-10.47%) 3.28s 3.30s p=0.004 n=6
Bind Time 1.02s (± 0.40%) 0.97s (± 0.78%) 🟩-0.05s (- 4.91%) 0.96s 0.98s p=0.003 n=6
Check Time 18.21s (± 0.67%) 17.91s (± 0.37%) -0.30s (- 1.66%) 17.81s 17.97s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.91s (± 0.54%) 22.17s (± 0.26%) 🟩-0.74s (- 3.22%) 22.08s 22.23s p=0.005 n=6
xstate - node (v16.17.1, x64)
Memory used 546,665k (± 0.01%) 544,187k (± 0.02%) -2,477k (- 0.45%) 544,121k 544,373k p=0.005 n=6
Parse Time 4.77s (± 0.22%) 4.22s (± 0.30%) 🟩-0.55s (-11.47%) 4.20s 4.23s p=0.004 n=6
Bind Time 1.84s (± 0.28%) 1.73s (± 0.44%) 🟩-0.11s (- 6.24%) 1.72s 1.74s p=0.004 n=6
Check Time 3.06s (± 0.65%) 2.97s (± 0.56%) -0.09s (- 2.94%) 2.94s 2.99s p=0.005 n=6
Emit Time 0.09s (± 4.45%) 0.09s (± 0.00%) ~ 0.09s 0.09s p=0.405 n=6
Total Time 9.76s (± 0.29%) 9.01s (± 0.33%) 🟩-0.75s (- 7.68%) 8.97s 9.04s p=0.005 n=6
Angular - node (v14.15.1, x64)
Memory used 352,818k (± 0.01%) 351,944k (± 0.00%) -874k (- 0.25%) 351,918k 351,956k p=0.005 n=6
Parse Time 4.23s (± 0.35%) 3.74s (± 0.53%) 🟩-0.49s (-11.55%) 3.72s 3.77s p=0.005 n=6
Bind Time 1.28s (± 0.80%) 1.25s (± 0.51%) -0.03s (- 2.60%) 1.24s 1.26s p=0.004 n=6
Check Time 9.74s (± 0.71%) 9.66s (± 0.33%) ~ 9.62s 9.70s p=0.065 n=6
Emit Time 8.37s (± 0.54%) 8.20s (± 0.64%) -0.17s (- 2.01%) 8.16s 8.30s p=0.006 n=6
Total Time 23.61s (± 0.42%) 22.85s (± 0.22%) 🟩-0.77s (- 3.24%) 22.77s 22.92s p=0.005 n=6
Compiler-Unions - node (v14.15.1, x64)
Memory used 188,784k (± 0.05%) 188,435k (± 0.01%) -348k (- 0.18%) 188,417k 188,455k p=0.005 n=6
Parse Time 1.83s (± 0.41%) 1.59s (± 0.51%) 🟩-0.25s (-13.38%) 1.58s 1.60s p=0.004 n=6
Bind Time 0.87s (± 0.97%) 0.85s (± 0.61%) -0.02s (- 2.12%) 0.84s 0.85s p=0.004 n=6
Check Time 10.50s (± 0.59%) 10.26s (± 0.37%) -0.24s (- 2.25%) 10.19s 10.29s p=0.005 n=6
Emit Time 3.39s (± 3.89%) 3.08s (± 0.93%) 🟩-0.31s (- 9.14%) 3.04s 3.12s p=0.005 n=6
Total Time 16.59s (± 1.03%) 15.78s (± 0.32%) 🟩-0.81s (- 4.89%) 15.70s 15.85s p=0.005 n=6
Monaco - node (v14.15.1, x64)
Memory used 337,993k (± 0.00%) 337,278k (± 0.01%) -715k (- 0.21%) 337,262k 337,310k p=0.005 n=6
Parse Time 3.20s (± 0.67%) 2.87s (± 0.36%) 🟩-0.33s (-10.16%) 2.86s 2.89s p=0.005 n=6
Bind Time 1.11s (± 0.46%) 1.12s (± 0.49%) ~ 1.11s 1.12s p=0.640 n=6
Check Time 8.11s (± 0.56%) 7.96s (± 0.81%) -0.15s (- 1.85%) 7.86s 8.03s p=0.005 n=6
Emit Time 4.80s (± 0.83%) 4.55s (± 0.98%) 🟩-0.25s (- 5.27%) 4.49s 4.60s p=0.005 n=6
Total Time 17.23s (± 0.34%) 16.50s (± 0.51%) 🟩-0.73s (- 4.22%) 16.33s 16.56s p=0.005 n=6
TFS - node (v14.15.1, x64)
Memory used 294,575k (± 0.01%) 293,924k (± 0.01%) -651k (- 0.22%) 293,884k 293,941k p=0.005 n=6
Parse Time 2.74s (± 0.68%) 2.42s (± 0.48%) 🟩-0.32s (-11.78%) 2.41s 2.44s p=0.005 n=6
Bind Time 1.11s (± 0.95%) 1.08s (± 0.50%) -0.02s (- 1.81%) 1.08s 1.09s p=0.008 n=6
Check Time 7.51s (± 0.47%) 7.45s (± 0.40%) -0.07s (- 0.93%) 7.41s 7.48s p=0.013 n=6
Emit Time 4.63s (± 1.03%) 4.29s (± 1.01%) 🟩-0.34s (- 7.31%) 4.21s 4.33s p=0.005 n=6
Total Time 15.99s (± 0.44%) 15.24s (± 0.43%) 🟩-0.75s (- 4.69%) 15.12s 15.31s p=0.005 n=6
material-ui - node (v14.15.1, x64)
Memory used 471,622k (± 0.00%) 470,749k (± 0.01%) -873k (- 0.19%) 470,722k 470,778k p=0.005 n=6
Parse Time 3.88s (± 0.72%) 3.44s (± 0.61%) 🟩-0.44s (-11.26%) 3.42s 3.48s p=0.005 n=6
Bind Time 1.05s (± 0.00%) 1.01s (± 0.40%) 🟩-0.04s (- 3.97%) 1.00s 1.01s p=0.002 n=6
Check Time 19.14s (± 0.77%) 18.92s (± 0.31%) -0.22s (- 1.16%) 18.82s 18.99s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 24.07s (± 0.57%) 23.37s (± 0.23%) -0.70s (- 2.89%) 23.27s 23.42s p=0.005 n=6
xstate - node (v14.15.1, x64)
Memory used 534,749k (± 0.00%) 533,029k (± 0.02%) -1,721k (- 0.32%) 532,837k 533,091k p=0.005 n=6
Parse Time 5.12s (± 0.82%) 4.36s (± 0.23%) 🟩-0.76s (-14.87%) 4.35s 4.37s p=0.005 n=6
Bind Time 1.71s (± 0.60%) 1.76s (± 0.78%) +0.06s (+ 3.42%) 1.74s 1.78s p=0.004 n=6
Check Time 3.13s (± 0.48%) 3.14s (± 0.70%) ~ 3.11s 3.17s p=0.256 n=6
Emit Time 0.10s (± 4.01%) 0.09s (± 0.00%) 🟩-0.01s (-11.48%) 0.09s 0.09s p=0.002 n=6
Total Time 10.06s (± 0.36%) 9.36s (± 0.34%) 🟩-0.70s (- 6.96%) 9.32s 9.40s p=0.005 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-135-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v18.10.0, x64)
  • node (v16.17.1, x64)
  • node (v14.15.1, x64)
Scenarios
  • Angular - node (v18.10.0, x64)
  • Angular - node (v16.17.1, x64)
  • Angular - node (v14.15.1, x64)
  • Compiler-Unions - node (v18.10.0, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Compiler-Unions - node (v14.15.1, x64)
  • Monaco - node (v18.10.0, x64)
  • Monaco - node (v16.17.1, x64)
  • Monaco - node (v14.15.1, x64)
  • TFS - node (v18.10.0, x64)
  • TFS - node (v16.17.1, x64)
  • TFS - node (v14.15.1, x64)
  • material-ui - node (v18.10.0, x64)
  • material-ui - node (v16.17.1, x64)
  • material-ui - node (v14.15.1, x64)
  • xstate - node (v18.10.0, x64)
  • xstate - node (v16.17.1, x64)
  • xstate - node (v14.15.1, x64)
Benchmark Name Iterations
Current 52656 6
Baseline main 6

TSServer

Comparison Report - main..52656
Metric main 52656 Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 2,496ms (± 1.28%) 2,378ms (± 1.33%) 🟩-118ms (- 4.73%) 2,338ms 2,422ms p=0.005 n=6
Req 2 - geterr 5,511ms (± 0.57%) 5,413ms (± 0.51%) -98ms (- 1.77%) 5,375ms 5,452ms p=0.005 n=6
Req 3 - references 370ms (± 1.13%) 345ms (± 1.33%) 🟩-25ms (- 6.77%) 340ms 353ms p=0.005 n=6
Req 4 - navto 284ms (± 0.99%) 277ms (± 0.90%) -7ms (- 2.52%) 272ms 279ms p=0.005 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 73ms (± 1.03%) 69ms (± 5.72%) ~ 67ms 77ms p=0.061 n=6
CompilerTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 2,739ms (± 0.79%) 2,494ms (± 0.89%) 🟩-245ms (- 8.93%) 2,459ms 2,522ms p=0.005 n=6
Req 2 - geterr 4,059ms (± 0.64%) 3,990ms (± 0.63%) -69ms (- 1.70%) 3,959ms 4,035ms p=0.008 n=6
Req 3 - references 368ms (± 0.72%) 362ms (± 0.69%) -6ms (- 1.63%) 359ms 366ms p=0.013 n=6
Req 4 - navto 293ms (± 0.88%) 293ms (± 0.91%) ~ 290ms 298ms p=1.000 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 66ms (± 0.79%) 66ms (± 9.84%) ~ 62ms 77ms p=0.366 n=6
xstateTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 3,490ms (± 0.82%) 3,181ms (± 0.70%) 🟩-309ms (- 8.85%) 3,145ms 3,204ms p=0.005 n=6
Req 2 - geterr 1,648ms (± 0.66%) 1,682ms (± 1.35%) +35ms (+ 2.09%) 1,657ms 1,711ms p=0.005 n=6
Req 3 - references 104ms (± 1.05%) 103ms (± 2.07%) ~ 101ms 106ms p=0.566 n=6
Req 4 - navto 355ms (± 0.53%) 367ms (± 0.93%) +13ms (+ 3.57%) 363ms 373ms p=0.005 n=6
Req 5 - completionInfo count 3,175 (± 0.00%) 3,175 (± 0.00%) ~ 3,175 3,175 p=1.000 n=6
Req 5 - completionInfo 450ms (± 0.90%) 445ms (± 1.17%) ~ 439ms 452ms p=0.092 n=6
Compiler-UnionsTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,770ms (± 0.58%) 2,529ms (± 0.91%) 🟩-242ms (- 8.72%) 2,496ms 2,554ms p=0.005 n=6
Req 2 - geterr 5,930ms (± 0.53%) 5,774ms (± 0.36%) -156ms (- 2.63%) 5,743ms 5,800ms p=0.005 n=6
Req 3 - references 377ms (± 0.76%) 360ms (± 0.49%) 🟩-18ms (- 4.64%) 357ms 362ms p=0.005 n=6
Req 4 - navto 277ms (± 0.91%) 265ms (± 0.67%) 🟩-12ms (- 4.33%) 263ms 268ms p=0.005 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 79ms (± 2.70%) 88ms (± 2.42%) +9ms (+11.37%) 84ms 90ms p=0.005 n=6
CompilerTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,924ms (± 0.37%) 2,654ms (± 1.00%) 🟩-270ms (- 9.23%) 2,615ms 2,692ms p=0.005 n=6
Req 2 - geterr 4,412ms (± 0.53%) 4,332ms (± 0.38%) -81ms (- 1.82%) 4,308ms 4,352ms p=0.005 n=6
Req 3 - references 382ms (± 1.07%) 376ms (± 0.48%) -6ms (- 1.66%) 373ms 378ms p=0.036 n=6
Req 4 - navto 287ms (± 0.56%) 284ms (± 0.36%) -4ms (- 1.22%) 282ms 285ms p=0.004 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 71ms (± 0.57%) 67ms (± 1.26%) 🟩-5ms (- 6.56%) 65ms 67ms p=0.003 n=6
xstateTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 3,626ms (± 0.35%) 3,341ms (± 0.52%) 🟩-285ms (- 7.86%) 3,319ms 3,362ms p=0.005 n=6
Req 2 - geterr 1,825ms (± 1.51%) 2,014ms (± 2.38%) +189ms (+10.37%) 1,923ms 2,049ms p=0.005 n=6
Req 3 - references 113ms (± 1.42%) 110ms (± 2.07%) -3ms (- 2.51%) 106ms 112ms p=0.034 n=6
Req 4 - navto 344ms (± 1.02%) 355ms (± 0.95%) +11ms (+ 3.15%) 349ms 358ms p=0.006 n=6
Req 5 - completionInfo count 3,175 (± 0.00%) 3,175 (± 0.00%) ~ 3,175 3,175 p=1.000 n=6
Req 5 - completionInfo 457ms (± 0.99%) 450ms (± 1.43%) ~ 443ms 462ms p=0.064 n=6
Compiler-UnionsTSServer - node (v14.15.1, x64)
Req 1 - updateOpen 2,884ms (± 0.28%) 2,589ms (± 0.51%) 🟩-295ms (-10.22%) 2,572ms 2,604ms p=0.005 n=6
Req 2 - geterr 6,277ms (± 0.73%) 6,207ms (± 0.59%) -70ms (- 1.11%) 6,156ms 6,250ms p=0.020 n=6
Req 3 - references 391ms (± 0.53%) 368ms (± 0.48%) 🟩-23ms (- 5.85%) 365ms 370ms p=0.005 n=6
Req 4 - navto 280ms (± 1.31%) 279ms (± 0.67%) ~ 278ms 283ms p=0.933 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 99ms (± 1.69%) 98ms (± 2.29%) ~ 94ms 100ms p=0.676 n=6
CompilerTSServer - node (v14.15.1, x64)
Req 1 - updateOpen 3,038ms (± 0.96%) 2,772ms (± 0.97%) 🟩-266ms (- 8.75%) 2,735ms 2,811ms p=0.005 n=6
Req 2 - geterr 4,695ms (± 0.40%) 4,642ms (± 1.86%) ~ 4,468ms 4,700ms p=0.093 n=6
Req 3 - references 404ms (± 0.43%) 388ms (± 0.53%) 🟩-16ms (- 3.96%) 386ms 391ms p=0.005 n=6
Req 4 - navto 293ms (± 1.30%) 287ms (± 1.15%) -6ms (- 1.94%) 284ms 293ms p=0.035 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 82ms (± 3.84%) 82ms (± 7.67%) ~ 79ms 95ms p=0.230 n=6
xstateTSServer - node (v14.15.1, x64)
Req 1 - updateOpen 3,955ms (± 0.60%) 3,628ms (± 0.52%) 🟩-327ms (- 8.27%) 3,600ms 3,655ms p=0.005 n=6
Req 2 - geterr 1,776ms (± 1.68%) 1,804ms (± 0.78%) ~ 1,785ms 1,825ms p=0.066 n=6
Req 3 - references 133ms (± 1.99%) 124ms (± 1.56%) 🟩-9ms (- 6.88%) 121ms 126ms p=0.005 n=6
Req 4 - navto 375ms (± 0.95%) 383ms (± 0.47%) +8ms (+ 2.13%) 381ms 386ms p=0.005 n=6
Req 5 - completionInfo count 3,175 (± 0.00%) 3,175 (± 0.00%) ~ 3,175 3,175 p=1.000 n=6
Req 5 - completionInfo 470ms (± 2.40%) 461ms (± 0.60%) ~ 457ms 465ms p=0.297 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-135-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v18.10.0, x64)
  • node (v16.17.1, x64)
  • node (v14.15.1, x64)
Scenarios
  • Compiler-UnionsTSServer - node (v18.10.0, x64)
  • Compiler-UnionsTSServer - node (v16.17.1, x64)
  • Compiler-UnionsTSServer - node (v14.15.1, x64)
  • CompilerTSServer - node (v18.10.0, x64)
  • CompilerTSServer - node (v16.17.1, x64)
  • CompilerTSServer - node (v14.15.1, x64)
  • xstateTSServer - node (v18.10.0, x64)
  • xstateTSServer - node (v16.17.1, x64)
  • xstateTSServer - node (v14.15.1, x64)
Benchmark Name Iterations
Current 52656 6
Baseline main 6

Startup

Comparison Report - main..52656
Metric main 52656 Delta Best Worst p-value
tsc-startup - node (v16.17.1, x64)
Execution time 142.53ms (± 0.21%) 140.66ms (± 0.18%) -1.87ms (- 1.31%) 140.03ms 143.81ms p=0.000 n=600
tsserver-startup - node (v16.17.1, x64)
Execution time 227.47ms (± 0.18%) 223.95ms (± 0.18%) -3.52ms (- 1.55%) 222.71ms 227.24ms p=0.000 n=600
tsserverlibrary-startup - node (v16.17.1, x64)
Execution time 228.98ms (± 0.13%) 228.47ms (± 0.41%) -0.51ms (- 0.22%) 225.12ms 239.95ms p=0.000 n=600
typescript-startup - node (v16.17.1, x64)
Execution time 209.74ms (± 0.15%) 209.00ms (± 0.37%) -0.74ms (- 0.35%) 206.25ms 214.83ms p=0.000 n=600
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-135-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v16.17.1, x64)
Scenarios
  • tsc-startup - node (v16.17.1, x64)
  • tsserver-startup - node (v16.17.1, x64)
  • tsserverlibrary-startup - node (v16.17.1, x64)
  • typescript-startup - node (v16.17.1, x64)
Benchmark Name Iterations
Current 52656 6
Baseline main 6

Developer Information:

Download Benchmark

@Andarist
Copy link
Contributor

Andarist commented Feb 7, 2023

Not bad results 😉

@jakebailey
Copy link
Member Author

Yeah, though this was already known when we merged modules and gave up var downleveling. Really unfortunate that this helps so much; I'm trying to figure out if there's some subset of variables that can be changed or if it's just a million papercuts.

@Andarist
Copy link
Contributor

Andarist commented Feb 7, 2023

What’s the concern here though? Just the build time?

@jakebailey
Copy link
Member Author

jakebailey commented Feb 7, 2023

Well, I wouldn't merge this as-is because (in no particular order):

  • It introduces another third-party dep into the critical build chain, which right now is solely tsc and esbuild, and we have the ability to drop the latter if needed.
    • Though, arguably, the latter wouldn't be great for performance either, so what's the point in avoiding it?
  • It slows down the build, and I don't really want to re-introduce the idea of a "release" build of TypeScript.
    • But, we can just keep let/const for tests and catch violations not already caught by TS that way.
  • Loops get rewritten, which makes the files annoying to use while debugging; preferrably there would be a performance-focused variant of the transform which only did the "easy" ones.
    • Again, could be non-test only.
  • babel deoptimizes the files because they're so big, which effectively makes them as hard to patch as minified versions of TypeScript, and we have an unreasonable number of people who do that. I half-mitigated this by rerunning the output through esbuild, though that was to try and avoid the perf diff showing a startup speed boots from the simple fact that the files are smaller.
    • But, it could be implemented in esbuild directly.
  • Long term, I think it'd be bad to be ESM but have export var (which I believe is slower for consumers because they are live bindings and not const).

@jakebailey
Copy link
Member Author

It slows down the build

On the whole though, the "big" build tasks don't actually get longer because esbuild/babel are running in parallel with dts emit / bundling, but the actual bundle tasks go from half a second to 8 seconds, which isn't great for the debugging loop.

@fatcerberus
Copy link

fatcerberus commented Feb 7, 2023

I assume the performance hit for let/const vs. var is down to TDZ tracking. Which is unfortunate because var has enough footguns that I’d never use it directly in a codebase and I’m not running babel over my code as a build step just to change all the lets to vars.

@jakebailey
Copy link
Member Author

Yep, it's TDZ. I'm trying to figure out if there's a limited subset that can be changed to make parsing faster but it's not really turning out well.

@fatcerberus
Copy link

It’s weird because most potential TDZ violations can be detected syntactically so engines should have figured out how to optimize it away by now.

@jakebailey
Copy link
Member Author

I would certainly think. But at this point I'm having a hard time justifying not doing some form of this change.

@typescript-bot typescript-bot added Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels Feb 7, 2023
@jakebailey
Copy link
Member Author

jakebailey commented Feb 7, 2023

If people are stumbling on this, last time I tested this, it was 5-9%, and now it's more like 7-15%; the relative proportion has changed because we've increased the performance of the TS compiler since modules has merged meaning the fixed perf bump here is now a bigger overall proportion.

@jakebailey
Copy link
Member Author

I hacked up TypeScript and ts-perf to run this PR against bun v0.5.5. It doesn't seem like the engine Bun uses (Apple's JavaScriptCore) is affected by the TDZ problem; I'd chalk up any perf differences below to noise.

Note that these benchmark results are not comparable to the ones earlier in this thread; I performed these tests on an entirely different perf-tuned machine in my office, not the corp-owned machine that we normally use. But, I probably will try and upstream my hacks to ts-perf as I've been wanting to test some more interesting hosts.

Details

Loading benchmark and comparing to baseline.
Comparison Report - main..HEAD

Metric main HEAD Delta Best Worst p-value
Angular - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.57s (± 0.53%) 2.53s (± 2.39%) ~ 2.49s 2.65s p=0.064 n=6
Bind Time 1.05s (± 0.85%) 1.05s (± 1.55%) ~ 1.03s 1.08s p=0.801 n=6
Check Time 9.85s (± 0.58%) 9.81s (± 0.49%) ~ 9.76s 9.89s p=0.261 n=6
Emit Time 8.69s (± 0.51%) 8.58s (± 0.49%) -0.11s (- 1.27%) 8.51s 8.63s p=0.008 n=6
Total Time 22.15s (± 0.31%) 21.98s (± 0.54%) -0.18s (- 0.80%) 21.82s 22.17s p=0.013 n=6
Compiler - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 1.47s (± 0.93%) 1.51s (± 1.37%) +0.03s (+ 2.26%) 1.49s 1.54s p=0.009 n=6
Bind Time 1.03s (± 1.13%) 1.04s (± 1.16%) ~ 1.02s 1.05s p=0.131 n=6
Check Time 8.49s (± 0.72%) 8.45s (± 0.40%) ~ 8.41s 8.49s p=0.332 n=6
Emit Time 3.68s (± 0.96%) 3.61s (± 0.65%) -0.07s (- 1.81%) 3.59s 3.64s p=0.005 n=6
Total Time 14.67s (± 0.53%) 14.62s (± 0.30%) ~ 14.57s 14.68s p=0.297 n=6
Compiler-Unions - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 1.43s (± 2.27%) 1.41s (± 2.00%) ~ 1.36s 1.43s p=0.346 n=6
Bind Time 0.98s (± 0.77%) 0.96s (± 2.68%) ~ 0.94s 1.01s p=0.191 n=6
Check Time 11.62s (± 0.89%) 11.69s (± 1.19%) ~ 11.50s 11.85s p=0.377 n=6
Emit Time 3.44s (± 1.07%) 3.39s (± 1.48%) ~ 3.33s 3.47s p=0.106 n=6
Total Time 17.46s (± 0.46%) 17.45s (± 0.72%) ~ 17.29s 17.63s p=0.936 n=6
Monaco - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.41s (± 2.65%) 2.35s (± 3.28%) -0.06s (- 2.35%) 2.31s 2.51s p=0.044 n=6
Bind Time 1.35s (± 1.89%) 1.36s (± 1.90%) ~ 1.32s 1.39s p=0.872 n=6
Check Time 8.69s (± 0.35%) 8.67s (± 0.31%) ~ 8.65s 8.72s p=0.406 n=6
Emit Time 4.75s (± 1.27%) 4.72s (± 1.57%) ~ 4.64s 4.84s p=0.376 n=6
Total Time 17.20s (± 0.65%) 17.10s (± 1.05%) ~ 16.96s 17.45s p=0.090 n=6
TFS - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.08s (± 1.41%) 2.05s (± 1.29%) ~ 2.03s 2.10s p=0.106 n=6
Bind Time 1.16s (± 2.59%) 1.17s (± 1.13%) ~ 1.15s 1.19s p=0.220 n=6
Check Time 8.04s (± 0.36%) 8.08s (± 0.52%) ~ 8.04s 8.15s p=0.087 n=6
Emit Time 4.35s (± 0.55%) 4.27s (± 0.84%) -0.08s (- 1.80%) 4.21s 4.32s p=0.008 n=6
Total Time 15.62s (± 0.18%) 15.58s (± 0.34%) ~ 15.51s 15.65s p=0.173 n=6
material-ui - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.75s (± 1.58%) 2.75s (± 2.03%) ~ 2.71s 2.86s p=0.684 n=6
Bind Time 0.92s (± 3.60%) 0.94s (± 0.55%) ~ 0.94s 0.95s p=0.210 n=6
Check Time 21.98s (± 1.38%) 21.26s (± 0.65%) 🟩-0.72s (- 3.27%) 21.00s 21.40s p=0.008 n=6
Emit Time 0.00s (±244.70%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=0.405 n=6
Total Time 25.66s (± 1.32%) 24.96s (± 0.55%) -0.70s (- 2.72%) 24.70s 25.09s p=0.020 n=6
xstate - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 4.15s (± 0.72%) 4.03s (± 0.30%) 🟩-0.13s (- 3.01%) 4.01s 4.04s p=0.005 n=6
Bind Time 1.58s (± 0.89%) 1.61s (± 0.94%) +0.03s (+ 1.69%) 1.59s 1.63s p=0.018 n=6
Check Time 4.03s (± 0.44%) 4.08s (± 0.51%) +0.05s (+ 1.24%) 4.06s 4.11s p=0.010 n=6
Emit Time 0.23s (± 2.28%) 0.22s (± 0.00%) -0.01s (- 2.94%) 0.22s 0.22s p=0.025 n=6
Total Time 9.99s (± 0.32%) 9.94s (± 0.24%) -0.05s (- 0.52%) 9.90s 9.96s p=0.023 n=6
System
Machine Nameelcheapo
Platformlinux 6.1.9-arch1-2
Architecturex64
Available Memory8 GB
Available Memory5 GB
CPUs6 × Intel(R) Core(TM) i5-8400 CPU @ 2.80GHz
Hosts
  • bun (v0.5.5, x64)
Scenarios
  • Angular - bun (v0.5.5, x64)
  • Compiler - bun (v0.5.5, x64)
  • Compiler-Unions - bun (v0.5.5, x64)
  • Monaco - bun (v0.5.5, x64)
  • TFS - bun (v0.5.5, x64)
  • material-ui - bun (v0.5.5, x64)
  • xstate - bun (v0.5.5, x64)
Benchmark Name Iterations
Current HEAD 6
Baseline main 6

@jakebailey
Copy link
Member Author

But, i'm going to rerun this just to be sure I didn't mess up somehow.

@jakebailey
Copy link
Member Author

jakebailey commented Feb 8, 2023

Nah, seems correct. A second go:

Details

Loading benchmark and comparing to baseline.
Comparison Report - main..HEAD

Metric main HEAD Delta Best Worst p-value
Angular - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.57s (± 0.98%) 2.54s (± 2.64%) ~ 2.50s 2.67s p=0.062 n=6
Bind Time 1.06s (± 1.10%) 1.07s (± 3.01%) ~ 1.04s 1.13s p=0.806 n=6
Check Time 9.80s (± 0.54%) 9.84s (± 0.68%) ~ 9.77s 9.96s p=0.573 n=6
Emit Time 8.71s (± 1.07%) 8.62s (± 0.93%) ~ 8.55s 8.76s p=0.128 n=6
Total Time 22.14s (± 0.37%) 22.07s (± 0.71%) ~ 21.86s 22.30s p=0.471 n=6
Compiler - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 1.47s (± 1.27%) 1.51s (± 1.62%) +0.04s (+ 2.95%) 1.49s 1.55s p=0.011 n=6
Bind Time 1.03s (± 1.50%) 1.03s (± 1.84%) ~ 1.01s 1.06s p=0.799 n=6
Check Time 8.44s (± 0.59%) 8.47s (± 0.66%) ~ 8.41s 8.56s p=0.261 n=6
Emit Time 3.67s (± 0.80%) 3.63s (± 0.73%) ~ 3.59s 3.66s p=0.077 n=6
Total Time 14.60s (± 0.53%) 14.65s (± 0.44%) ~ 14.56s 14.73s p=0.298 n=6
Compiler-Unions - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 1.42s (± 1.06%) 1.41s (± 3.28%) ~ 1.37s 1.49s p=0.368 n=6
Bind Time 0.97s (± 0.53%) 0.96s (± 1.70%) ~ 0.94s 0.98s p=0.542 n=6
Check Time 11.66s (± 0.40%) 11.63s (± 0.91%) ~ 11.47s 11.77s p=0.810 n=6
Emit Time 3.43s (± 1.19%) 3.35s (± 1.50%) -0.08s (- 2.38%) 3.29s 3.40s p=0.015 n=6
Total Time 17.47s (± 0.38%) 17.36s (± 0.40%) -0.12s (- 0.67%) 17.25s 17.43s p=0.019 n=6
Monaco - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.39s (± 0.57%) 2.35s (± 3.22%) ~ 2.30s 2.50s p=0.060 n=6
Bind Time 1.35s (± 5.27%) 1.33s (± 7.58%) ~ 1.13s 1.40s p=0.870 n=6
Check Time 8.70s (± 1.09%) 8.70s (± 0.83%) ~ 8.64s 8.83s p=0.809 n=6
Emit Time 4.75s (± 0.77%) 4.71s (± 0.46%) ~ 4.68s 4.74s p=0.072 n=6
Total Time 17.18s (± 0.42%) 17.08s (± 0.53%) ~ 17.01s 17.22s p=0.065 n=6
TFS - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.07s (± 0.66%) 2.05s (± 1.91%) ~ 2.03s 2.13s p=0.075 n=6
Bind Time 1.17s (± 2.28%) 1.17s (± 0.44%) ~ 1.17s 1.18s p=0.667 n=6
Check Time 8.02s (± 0.38%) 8.08s (± 0.26%) +0.06s (+ 0.79%) 8.05s 8.10s p=0.006 n=6
Emit Time 4.33s (± 1.72%) 4.26s (± 1.05%) ~ 4.20s 4.32s p=0.054 n=6
Total Time 15.58s (± 0.57%) 15.56s (± 0.37%) ~ 15.49s 15.65s p=0.873 n=6
material-ui - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.75s (± 0.61%) 2.76s (± 2.06%) ~ 2.72s 2.87s p=0.806 n=6
Bind Time 0.93s (± 4.07%) 0.95s (± 0.79%) ~ 0.94s 0.96s p=0.081 n=6
Check Time 22.12s (± 0.61%) 21.37s (± 0.38%) 🟩-0.75s (- 3.39%) 21.27s 21.51s p=0.005 n=6
Emit Time 0.00s (±244.70%) 0.00s (±244.70%) ~ 0.00s 0.01s p=1.000 n=6
Total Time 25.80s (± 0.63%) 25.09s (± 0.40%) -0.71s (- 2.75%) 24.97s 25.23s p=0.005 n=6
xstate - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 4.15s (± 0.20%) 4.02s (± 0.72%) 🟩-0.13s (- 3.10%) 3.99s 4.07s p=0.005 n=6
Bind Time 1.57s (± 1.32%) 1.63s (± 1.08%) +0.05s (+ 3.17%) 1.60s 1.64s p=0.006 n=6
Check Time 4.03s (± 0.30%) 4.07s (± 0.39%) +0.04s (+ 0.95%) 4.06s 4.10s p=0.005 n=6
Emit Time 0.23s (± 3.88%) 0.22s (± 1.84%) ~ 0.22s 0.23s p=0.086 n=6
Total Time 9.99s (± 0.22%) 9.94s (± 0.45%) -0.05s (- 0.50%) 9.89s 10.01s p=0.044 n=6
System
Machine Nameelcheapo
Platformlinux 6.1.9-arch1-2
Architecturex64
Available Memory8 GB
Available Memory5 GB
CPUs6 × Intel(R) Core(TM) i5-8400 CPU @ 2.80GHz
Hosts
  • bun (v0.5.5, x64)
Scenarios
  • Angular - bun (v0.5.5, x64)
  • Compiler - bun (v0.5.5, x64)
  • Compiler-Unions - bun (v0.5.5, x64)
  • Monaco - bun (v0.5.5, x64)
  • TFS - bun (v0.5.5, x64)
  • material-ui - bun (v0.5.5, x64)
  • xstate - bun (v0.5.5, x64)
Benchmark Name Iterations
Current HEAD 6
Baseline main 6

EDIT: Updated this table to say "bun"; I did that in the last one but forgot in this one. ts-perf doesn't currently support bun so this is a hack.

@Jarred-Sumner
Copy link

Jarred-Sumner commented Feb 8, 2023

Note that Bun runs the transpiler on every file loaded either via import or require (not eval, not new Function, etc). It doesn’t convert let/const variables to var. I wouldn’t expect the results to be different l. it might not work, but you could try BUN_DISABLE_TRANSPILER=1 & it would rule out the possibility of Bun’s transpiler changing the output before being evaluated

@jakebailey
Copy link
Member Author

I'm not sure what you mean; the comparison is main versus this PR, i.e. let/const versus var. So if they don't do this transformation (as you say), then the comparison shows that their engine doesn't have this TDZ performance hit, at least not to any great extent.

@jakebailey
Copy link
Member Author

Unless you mean that I should try disabling the transpiler to make sure 100% they don't already do this transformation.

@Jarred-Sumner
Copy link

Jarred-Sumner commented Feb 8, 2023

The environment variable which prevents the transpiler from running is here:

https://github.com/oven-sh/bun/blob/main/src/bundler.zig#L539

I think the results were already correct, just that since it’s two transpilers running it’s harder to be 100% sure. I sent the above message without re-reading the words I wrote very carefully. Bun doesn’t convert let or const to var, but does a bunch of other stuff. I do know that ESM imports bindings have virtually no overhead in JSC (there was a thread in Node’s issues about ESM imports being slower than var at some point)

@marvinhagemeister
Copy link

Looping in @evanw because that may be an interesting optimization to do right in esbuild itself.

@jakebailey
Copy link
Member Author

Looping in @evanw because that may be an interesting optimization to do right in esbuild itself.

See: evanw/esbuild#2889 (comment)

@jakebailey

This comment was marked as outdated.

@typescript-bot

This comment was marked as outdated.

@typescript-bot

This comment was marked as outdated.

@jakebailey

This comment was marked as outdated.

@typescript-bot

This comment was marked as outdated.

@jakebailey
Copy link
Member Author

@typescript-bot new perf test bun

@typescript-bot
Copy link
Collaborator

typescript-bot commented Aug 30, 2023

Heya @jakebailey, I've started to run the bun perf test suite on this PR at e67fbc0. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - bun (v0.7.3, x64)
Memory used 321,810k (± 0.23%) 322,091k (± 0.28%) ~ 321,114k 323,375k p=0.689 n=6
Parse Time 2.32s (± 0.72%) 2.35s (± 0.79%) +0.03s (+ 1.44%) 2.33s 2.38s p=0.012 n=6
Bind Time 0.95s (± 5.36%) 0.94s (± 5.06%) ~ 0.90s 1.01s p=0.687 n=6
Check Time 7.94s (± 1.68%) 8.04s (± 0.37%) ~ 8.00s 8.07s p=0.254 n=6
Emit Time 6.44s (± 0.42%) 6.47s (± 0.77%) ~ 6.42s 6.55s p=0.466 n=6
Total Time 17.65s (± 0.56%) 17.80s (± 0.38%) +0.15s (+ 0.84%) 17.73s 17.90s p=0.029 n=6
Compiler-Unions - bun (v0.7.3, x64)
Memory used 246,169k (± 3.04%) 243,993k (± 2.10%) ~ 237,438k 249,524k p=0.575 n=6
Parse Time 1.26s (± 0.71%) 1.28s (± 1.74%) ~ 1.25s 1.31s p=0.164 n=6
Bind Time 0.85s (± 2.53%) 0.85s (± 0.96%) ~ 0.84s 0.86s p=0.682 n=6
Check Time 8.90s (± 0.23%) 9.00s (± 0.71%) ~ 8.88s 9.06s p=0.053 n=6
Emit Time 2.76s (± 0.86%) 2.74s (± 0.64%) ~ 2.72s 2.77s p=0.072 n=6
Total Time 13.78s (± 0.29%) 13.86s (± 0.35%) +0.09s (+ 0.63%) 13.77s 13.91s p=0.015 n=6
Monaco - bun (v0.7.3, x64)
Memory used 370,241k (± 0.37%) 370,192k (± 0.28%) ~ 369,196k 371,592k p=1.000 n=6
Parse Time 2.06s (± 0.57%) 2.05s (± 0.85%) ~ 2.03s 2.08s p=0.084 n=6
Bind Time 1.11s (± 2.48%) 1.11s (± 2.26%) ~ 1.08s 1.14s p=0.807 n=6
Check Time 7.19s (± 0.65%) 7.21s (± 0.52%) ~ 7.17s 7.28s p=0.746 n=6
Emit Time 3.75s (± 1.08%) 3.76s (± 0.93%) ~ 3.73s 3.82s p=0.571 n=6
Total Time 14.11s (± 0.29%) 14.13s (± 0.51%) ~ 14.06s 14.26s p=0.936 n=6
TFS - bun (v0.7.3, x64)
Memory used 317,835k (± 0.13%) 317,724k (± 0.25%) ~ 316,647k 319,042k p=0.378 n=6
Parse Time 1.80s (± 0.67%) 1.79s (± 0.55%) ~ 1.78s 1.80s p=0.138 n=6
Bind Time 1.00s (± 1.21%) 0.98s (± 1.78%) ~ 0.96s 1.01s p=0.059 n=6
Check Time 6.62s (± 0.64%) 6.68s (± 0.54%) +0.05s (+ 0.81%) 6.62s 6.73s p=0.045 n=6
Emit Time 3.44s (± 0.83%) 3.50s (± 1.23%) +0.06s (+ 1.79%) 3.45s 3.58s p=0.013 n=6
Total Time 12.88s (± 0.38%) 12.96s (± 0.53%) ~ 12.87s 13.05s p=0.065 n=6
material-ui - bun (v0.7.3, x64)
Memory used 537,607k (± 0.25%) 508,460k (± 7.12%) ~ 464,102k 543,543k p=0.471 n=6
Parse Time 2.50s (± 0.75%) 2.45s (± 0.71%) -0.04s (- 1.67%) 2.44s 2.48s p=0.007 n=6
Bind Time 0.71s (± 1.06%) 0.76s (± 3.70%) +0.05s (+ 6.82%) 0.73s 0.80s p=0.005 n=6
Check Time 15.76s (± 0.53%) 15.80s (± 0.77%) ~ 15.66s 15.99s p=0.423 n=6
Emit Time 0.00s (±244.70%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=0.405 n=6
Total Time 18.97s (± 0.41%) 19.02s (± 0.62%) ~ 18.88s 19.19s p=0.374 n=6
xstate - bun (v0.7.3, x64)
Memory used 471,655k (± 9.65%) 472,130k (± 8.15%) ~ 436,287k 509,641k p=1.000 n=6
Parse Time 3.60s (± 1.11%) 3.58s (± 0.38%) ~ 3.56s 3.60s p=0.627 n=6
Bind Time 1.41s (± 2.84%) 1.41s (± 1.17%) ~ 1.39s 1.43s p=0.870 n=6
Check Time 3.54s (± 0.81%) 3.62s (± 0.33%) +0.08s (+ 2.31%) 3.60s 3.63s p=0.004 n=6
Emit Time 0.22s (± 9.08%) 0.21s (± 3.55%) ~ 0.20s 0.22s p=0.652 n=6
Total Time 8.77s (± 0.51%) 8.83s (± 0.30%) +0.06s (+ 0.74%) 8.80s 8.86s p=0.029 n=6
System info unknown
Hosts
  • bun (v0.7.3, x64)
Scenarios
  • Angular - bun (v0.7.3, x64)
  • Compiler-Unions - bun (v0.7.3, x64)
  • Monaco - bun (v0.7.3, x64)
  • TFS - bun (v0.7.3, x64)
  • material-ui - bun (v0.7.3, x64)
  • xstate - bun (v0.7.3, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - bun (v0.7.3, x64)
Execution time 425.30ms (± 0.10%) 423.55ms (± 0.11%) -1.75ms (- 0.41%) 422.65ms 432.93ms p=0.000 n=600
tsserverlibrary-startup - bun (v0.7.3, x64)
Execution time 661.41ms (± 0.14%) 656.76ms (± 0.13%) -4.65ms (- 0.70%) 655.22ms 678.21ms p=0.000 n=600
typescript-startup - bun (v0.7.3, x64)
Execution time 662.40ms (± 0.11%) 654.86ms (± 0.14%) -7.54ms (- 1.14%) 653.45ms 672.29ms p=0.000 n=600
System info unknown
Hosts
  • bun (v0.7.3, x64)
Scenarios
  • tsc-startup - bun (v0.7.3, x64)
  • tsserverlibrary-startup - bun (v0.7.3, x64)
  • typescript-startup - bun (v0.7.3, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

@typescript-bot perf test public

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jan 11, 2024

Heya @jakebailey, I've started to run the public perf test suite on this PR at 0c249d9. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
mui-docs - node (v20.5.1, x64)
Memory used 1,735,248k (± 0.00%) 1,734,982k (± 0.00%) -266k (- 0.02%) 1,734,964k 1,735,005k p=0.005 n=6
Parse Time 6.83s (± 0.46%) 6.81s (± 0.44%) ~ 6.77s 6.85s p=0.333 n=6
Bind Time 2.42s (± 1.10%) 2.42s (± 0.56%) ~ 2.41s 2.44s p=0.459 n=6
Check Time 52.39s (± 0.40%) 52.55s (± 0.60%) ~ 51.97s 52.91s p=0.128 n=6
Emit Time 0.15s (± 0.00%) 0.15s (± 0.00%) ~ 0.15s 0.15s p=1.000 n=6
Total Time 61.78s (± 0.30%) 61.94s (± 0.51%) ~ 61.35s 62.28s p=0.149 n=6
self-build-src - node (v20.5.1, x64)
Memory used 2,751,144k (± 6.91%) 2,571,396k (± 0.03%) 🟩-179,748k (- 6.53%) 2,570,753k 2,573,098k p=0.005 n=6
Parse Time 5.05s (± 0.81%) 5.04s (± 0.58%) ~ 4.99s 5.07s p=0.748 n=6
Bind Time 1.99s (± 0.49%) 2.03s (± 1.35%) +0.04s (+ 1.84%) 1.99s 2.07s p=0.016 n=6
Check Time 32.12s (± 0.35%) 32.20s (± 0.16%) ~ 32.12s 32.26s p=0.298 n=6
Emit Time 2.79s (± 3.97%) 2.79s (± 4.04%) ~ 2.62s 2.94s p=0.936 n=6
Total Time 41.97s (± 0.22%) 42.07s (± 0.19%) ~ 41.97s 42.17s p=0.093 n=6
self-compiler - node (v20.5.1, x64)
Memory used 419,228k (± 0.01%) 419,355k (± 0.29%) ~ 418,746k 421,793k p=0.066 n=6
Parse Time 2.88s (± 0.48%) 2.87s (± 0.97%) ~ 2.84s 2.91s p=1.000 n=6
Bind Time 1.13s (± 0.74%) 1.14s (± 0.36%) ~ 1.13s 1.14s p=0.285 n=6
Check Time 14.12s (± 0.33%) 14.10s (± 0.41%) ~ 14.03s 14.17s p=0.688 n=6
Emit Time 1.05s (± 1.17%) 1.04s (± 0.72%) ~ 1.03s 1.05s p=1.000 n=6
Total Time 19.17s (± 0.33%) 19.15s (± 0.28%) ~ 19.08s 19.22s p=0.467 n=6
vscode - node (v20.5.1, x64)
Memory used 2,828,007k (± 0.01%) 2,827,290k (± 0.01%) -717k (- 0.03%) 2,827,184k 2,827,570k p=0.005 n=6
Parse Time 10.72s (± 0.17%) 10.71s (± 0.22%) ~ 10.67s 10.73s p=0.324 n=6
Bind Time 3.41s (± 0.82%) 3.45s (± 0.15%) +0.04s (+ 1.22%) 3.45s 3.46s p=0.010 n=6
Check Time 56.23s (± 0.45%) 56.64s (± 0.41%) +0.41s (+ 0.73%) 56.25s 56.88s p=0.031 n=6
Emit Time 16.81s (± 8.95%) 19.10s (± 7.87%) ~ 16.03s 19.77s p=0.297 n=6
Total Time 87.17s (± 1.84%) 89.90s (± 1.88%) ~ 86.47s 90.79s p=0.066 n=6
webpack - node (v20.5.1, x64)
Memory used 390,786k (± 0.01%) 390,445k (± 0.01%) -341k (- 0.09%) 390,395k 390,505k p=0.005 n=6
Parse Time 3.32s (± 0.40%) 3.30s (± 0.49%) ~ 3.28s 3.32s p=0.067 n=6
Bind Time 1.43s (± 0.82%) 1.44s (± 0.84%) ~ 1.42s 1.45s p=0.323 n=6
Check Time 12.78s (± 0.21%) 12.70s (± 0.31%) -0.08s (- 0.66%) 12.64s 12.74s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 17.54s (± 0.13%) 17.44s (± 0.18%) -0.10s (- 0.57%) 17.40s 17.47s p=0.005 n=6
System info unknown
Hosts
  • node (v20.5.1, x64)
Scenarios
  • mui-docs - node (v20.5.1, x64)
  • self-build-src - node (v20.5.1, x64)
  • self-compiler - node (v20.5.1, x64)
  • vscode - node (v20.5.1, x64)
  • webpack - node (v20.5.1, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 19, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,740k (± 0.01%) 295,106k (± 0.00%) -633k (- 0.21%) 295,088k 295,124k p=0.005 n=6
Parse Time 2.66s (± 0.19%) 2.62s (± 0.24%) -0.04s (- 1.63%) 2.61s 2.63s p=0.004 n=6
Bind Time 0.83s (± 0.66%) 0.83s (± 0.62%) +0.01s (+ 1.01%) 0.83s 0.84s p=0.038 n=6
Check Time 8.21s (± 0.30%) 8.06s (± 0.38%) -0.15s (- 1.79%) 8.01s 8.09s p=0.005 n=6
Emit Time 7.13s (± 0.71%) 6.98s (± 0.47%) -0.16s (- 2.20%) 6.95s 7.03s p=0.005 n=6
Total Time 18.83s (± 0.31%) 18.50s (± 0.30%) -0.33s (- 1.77%) 18.41s 18.57s p=0.005 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 193,791k (± 0.99%) 191,565k (± 0.06%) -2,226k (- 1.15%) 191,485k 191,801k p=0.005 n=6
Parse Time 1.36s (± 1.23%) 1.36s (± 2.25%) ~ 1.30s 1.38s p=0.681 n=6
Bind Time 0.72s (± 0.57%) 0.72s (± 0.57%) ~ 0.72s 0.73s p=1.000 n=6
Check Time 9.51s (± 0.82%) 9.31s (± 0.57%) -0.20s (- 2.10%) 9.24s 9.38s p=0.005 n=6
Emit Time 2.66s (± 0.46%) 2.64s (± 0.41%) ~ 2.63s 2.66s p=0.062 n=6
Total Time 14.24s (± 0.53%) 14.03s (± 0.58%) -0.21s (- 1.50%) 13.90s 14.13s p=0.005 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,366k (± 0.00%) 347,008k (± 0.01%) -358k (- 0.10%) 346,990k 347,042k p=0.005 n=6
Parse Time 2.48s (± 0.21%) 2.48s (± 0.59%) ~ 2.46s 2.50s p=0.451 n=6
Bind Time 0.93s (± 0.44%) 0.93s (± 0.44%) ~ 0.93s 0.94s p=0.218 n=6
Check Time 7.02s (± 0.28%) 7.04s (± 0.47%) ~ 7.01s 7.10s p=0.413 n=6
Emit Time 4.07s (± 0.34%) 4.06s (± 0.55%) ~ 4.03s 4.08s p=1.000 n=6
Total Time 14.49s (± 0.24%) 14.51s (± 0.26%) ~ 14.44s 14.54s p=0.332 n=6
TFS - node (v18.15.0, x64)
Memory used 302,727k (± 0.00%) 302,256k (± 0.01%) -471k (- 0.16%) 302,214k 302,312k p=0.005 n=6
Parse Time 2.01s (± 1.02%) 2.00s (± 0.55%) ~ 1.98s 2.01s p=0.503 n=6
Bind Time 1.00s (± 0.98%) 1.00s (± 0.82%) ~ 0.99s 1.01s p=0.394 n=6
Check Time 6.32s (± 0.60%) 6.30s (± 0.33%) ~ 6.27s 6.33s p=0.258 n=6
Emit Time 3.60s (± 0.48%) 3.54s (± 0.52%) -0.06s (- 1.53%) 3.52s 3.57s p=0.006 n=6
Total Time 12.93s (± 0.41%) 12.84s (± 0.17%) -0.09s (- 0.71%) 12.80s 12.86s p=0.008 n=6
material-ui - node (v18.15.0, x64)
Memory used 509,922k (± 0.00%) 509,549k (± 0.00%) -373k (- 0.07%) 509,531k 509,568k p=0.005 n=6
Parse Time 2.66s (± 0.41%) 2.70s (± 0.20%) +0.04s (+ 1.32%) 2.69s 2.70s p=0.004 n=6
Bind Time 0.98s (± 1.06%) 0.98s (± 1.00%) ~ 0.97s 0.99s p=0.673 n=6
Check Time 17.23s (± 0.39%) 17.26s (± 0.27%) ~ 17.18s 17.30s p=0.336 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 20.88s (± 0.38%) 20.93s (± 0.23%) ~ 20.86s 20.99s p=0.198 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,718,974k (± 0.00%) 1,718,497k (± 0.00%) -477k (- 0.03%) 1,718,463k 1,718,524k p=0.005 n=6
Parse Time 6.51s (± 0.40%) 6.58s (± 0.53%) +0.07s (+ 1.08%) 6.56s 6.65s p=0.005 n=6
Bind Time 2.36s (± 0.27%) 2.34s (± 0.66%) -0.02s (- 0.85%) 2.32s 2.36s p=0.024 n=6
Check Time 56.12s (± 0.36%) 56.40s (± 0.56%) ~ 55.92s 56.77s p=0.128 n=6
Emit Time 0.13s (± 0.00%) 0.14s (± 4.05%) ~ 0.13s 0.14s p=0.071 n=6
Total Time 65.12s (± 0.33%) 65.45s (± 0.53%) ~ 64.94s 65.89s p=0.093 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,394,975k (± 0.02%) 2,389,428k (± 0.02%) -5,547k (- 0.23%) 2,388,869k 2,390,500k p=0.005 n=6
Parse Time 4.98s (± 1.40%) 5.06s (± 1.19%) +0.08s (+ 1.64%) 4.98s 5.12s p=0.037 n=6
Bind Time 1.91s (± 1.32%) 1.88s (± 1.34%) ~ 1.85s 1.92s p=0.063 n=6
Check Time 33.57s (± 0.32%) 33.36s (± 0.21%) -0.21s (- 0.64%) 33.27s 33.45s p=0.013 n=6
Emit Time 2.71s (± 1.16%) 2.61s (± 1.03%) 🟩-0.10s (- 3.57%) 2.57s 2.64s p=0.005 n=6
Total Time 43.18s (± 0.38%) 42.93s (± 0.26%) -0.25s (- 0.58%) 42.73s 43.04s p=0.013 n=6
self-compiler - node (v18.15.0, x64)
Memory used 416,056k (± 0.01%) 415,592k (± 0.01%) -464k (- 0.11%) 415,533k 415,665k p=0.005 n=6
Parse Time 2.83s (± 0.68%) 2.74s (± 2.71%) 🟩-0.09s (- 3.06%) 2.64s 2.81s p=0.005 n=6
Bind Time 1.07s (± 0.48%) 1.11s (± 5.71%) ~ 1.07s 1.22s p=0.070 n=6
Check Time 15.33s (± 0.50%) 15.19s (± 0.39%) -0.14s (- 0.91%) 15.13s 15.25s p=0.010 n=6
Emit Time 1.14s (± 0.92%) 1.11s (± 0.57%) 🟩-0.03s (- 3.06%) 1.10s 1.12s p=0.004 n=6
Total Time 20.37s (± 0.38%) 20.15s (± 0.32%) -0.22s (- 1.10%) 20.07s 20.22s p=0.005 n=6
vscode - node (v18.15.0, x64)
Memory used 2,884,074k (± 0.00%) 2,883,374k (± 0.00%) -700k (- 0.02%) 2,883,258k 2,883,414k p=0.005 n=6
Parse Time 10.81s (± 0.26%) 10.78s (± 0.32%) ~ 10.74s 10.82s p=0.212 n=6
Bind Time 3.47s (± 0.30%) 3.48s (± 0.56%) ~ 3.46s 3.51s p=0.263 n=6
Check Time 61.41s (± 0.47%) 61.47s (± 0.34%) ~ 61.09s 61.69s p=0.872 n=6
Emit Time 16.44s (± 0.32%) 16.22s (± 0.49%) -0.22s (- 1.35%) 16.13s 16.34s p=0.005 n=6
Total Time 92.12s (± 0.30%) 91.94s (± 0.30%) ~ 91.47s 92.28s p=0.378 n=6
webpack - node (v18.15.0, x64)
Memory used 408,028k (± 0.01%) 407,754k (± 0.01%) -274k (- 0.07%) 407,690k 407,820k p=0.005 n=6
Parse Time 3.23s (± 0.23%) 3.20s (± 0.25%) -0.03s (- 0.88%) 3.19s 3.21s p=0.004 n=6
Bind Time 1.38s (± 0.30%) 1.40s (± 0.74%) +0.02s (+ 1.57%) 1.39s 1.42s p=0.004 n=6
Check Time 14.27s (± 0.20%) 14.23s (± 0.35%) ~ 14.19s 14.30s p=0.169 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 18.88s (± 0.19%) 18.84s (± 0.25%) ~ 18.79s 18.91s p=0.198 n=6
xstate - node (v18.15.0, x64)
Memory used 513,080k (± 0.01%) 512,180k (± 0.01%) -900k (- 0.18%) 512,089k 512,258k p=0.005 n=6
Parse Time 3.28s (± 0.37%) 3.25s (± 0.45%) -0.03s (- 0.81%) 3.23s 3.27s p=0.018 n=6
Bind Time 1.54s (± 0.35%) 1.55s (± 0.26%) ~ 1.55s 1.56s p=0.054 n=6
Check Time 2.86s (± 0.29%) 2.88s (± 0.36%) +0.03s (+ 0.99%) 2.87s 2.90s p=0.005 n=6
Emit Time 0.07s (± 7.03%) 0.07s (± 7.03%) ~ 0.07s 0.08s p=1.000 n=6
Total Time 7.75s (± 0.33%) 7.76s (± 0.16%) ~ 7.75s 7.78s p=0.518 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,336ms (± 0.78%) 2,337ms (± 0.78%) ~ 2,315ms 2,359ms p=1.000 n=6
Req 2 - geterr 5,794ms (± 0.37%) 5,677ms (± 0.53%) -117ms (- 2.01%) 5,656ms 5,738ms p=0.005 n=6
Req 3 - references 325ms (± 1.04%) 332ms (± 0.82%) +7ms (+ 2.00%) 330ms 337ms p=0.032 n=6
Req 4 - navto 272ms (± 0.19%) 265ms (± 0.15%) -7ms (- 2.63%) 265ms 266ms p=0.003 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 95ms (± 0.43%) 94ms (± 1.16%) ~ 93ms 96ms p=0.090 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,499ms (± 0.89%) 2,469ms (± 0.53%) -30ms (- 1.19%) 2,443ms 2,480ms p=0.016 n=6
Req 2 - geterr 4,264ms (± 0.33%) 4,159ms (± 0.49%) -105ms (- 2.45%) 4,132ms 4,184ms p=0.005 n=6
Req 3 - references 332ms (± 0.12%) 335ms (± 0.22%) +3ms (+ 0.90%) 334ms 336ms p=0.003 n=6
Req 4 - navto 283ms (± 0.45%) 278ms (± 0.44%) -6ms (- 1.94%) 276ms 279ms p=0.004 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 78ms (± 0.81%) 85ms (± 6.80%) 🔻+7ms (+ 8.33%) 78ms 90ms p=0.020 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,618ms (± 0.11%) 2,590ms (± 0.31%) -28ms (- 1.06%) 2,579ms 2,598ms p=0.005 n=6
Req 2 - geterr 1,725ms (± 1.98%) 1,720ms (± 1.64%) ~ 1,682ms 1,759ms p=0.689 n=6
Req 3 - references 115ms (±10.05%) 119ms (± 6.94%) ~ 102ms 123ms p=0.683 n=6
Req 4 - navto 371ms (± 0.20%) 375ms (± 0.90%) +4ms (+ 0.99%) 372ms 380ms p=0.011 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 308ms (± 1.42%) 308ms (± 1.41%) ~ 301ms 313ms p=0.517 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstateTSServer - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 153.76ms (± 0.17%) 151.55ms (± 0.17%) -2.20ms (- 1.43%) 150.52ms 154.90ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 228.85ms (± 0.16%) 229.69ms (± 0.19%) +0.84ms (+ 0.37%) 228.13ms 236.57ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 222.61ms (± 0.15%) 220.92ms (± 0.19%) -1.70ms (- 0.76%) 219.44ms 227.40ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 223.53ms (± 0.15%) 222.11ms (± 0.20%) -1.42ms (- 0.64%) 220.38ms 228.18ms p=0.000 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

@typescript-bot perf test this
@typescript-bot perf test this faster host=bun@1.1.17

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jun 30, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results
perf test this faster host=bun@1.1.17 ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-Unions - bun (v1.1.17, x64)
Errors 30 30 ~ ~ ~ p=1.000 n=6
Symbols 62,153 62,153 ~ ~ ~ p=1.000 n=6
Types 50,242 50,242 ~ ~ ~ p=1.000 n=6
Memory used 247,002k (± 4.81%) 255,207k (± 2.37%) ~ 251,606k 267,364k p=0.230 n=6
Parse Time 1.31s (± 1.33%) 1.31s (± 1.65%) ~ 1.29s 1.35s p=0.405 n=6
Bind Time 0.98s (± 1.50%) 0.97s (± 2.49%) ~ 0.94s 1.01s p=0.871 n=6
Check Time 10.22s (± 0.57%) 10.30s (± 0.93%) ~ 10.21s 10.43s p=0.077 n=6
Emit Time 3.43s (± 1.58%) 3.50s (± 1.07%) +0.06s (+ 1.89%) 3.44s 3.54s p=0.045 n=6
Total Time 15.94s (± 0.53%) 16.08s (± 0.46%) +0.14s (+ 0.87%) 15.97s 16.17s p=0.020 n=6
angular-1 - bun (v1.1.17, x64)
Errors 5 5 ~ ~ ~ p=1.000 n=6
Symbols 944,114 944,114 ~ ~ ~ p=1.000 n=6
Types 407,050 407,050 ~ ~ ~ p=1.000 n=6
Memory used 1,280,711k (± 0.16%) 1,280,677k (± 0.40%) ~ 1,270,826k 1,285,058k p=0.298 n=6
Parse Time 4.76s (± 0.42%) 4.74s (± 0.36%) ~ 4.72s 4.76s p=0.124 n=6
Bind Time 1.95s (± 0.43%) 1.96s (± 0.84%) ~ 1.93s 1.98s p=0.170 n=6
Check Time 29.56s (± 0.44%) 29.87s (± 0.84%) +0.31s (+ 1.05%) 29.62s 30.34s p=0.020 n=6
Emit Time 11.17s (± 1.13%) 11.41s (± 1.01%) +0.24s (+ 2.13%) 11.28s 11.57s p=0.016 n=6
Total Time 47.44s (± 0.45%) 47.97s (± 0.63%) +0.54s (+ 1.13%) 47.68s 48.48s p=0.013 n=6
mui-docs - bun (v1.1.17, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,133,039 2,133,039 ~ ~ ~ p=1.000 n=6
Types 926,444 926,444 ~ ~ ~ p=1.000 n=6
Memory used 2,313,268k (± 0.32%) 2,321,468k (± 0.42%) ~ 2,307,305k 2,332,902k p=0.230 n=6
Parse Time 5.52s (± 0.88%) 5.48s (± 0.34%) ~ 5.46s 5.51s p=0.295 n=6
Bind Time 1.54s (± 2.28%) 1.53s (± 3.33%) ~ 1.46s 1.61s p=0.518 n=6
Check Time 64.67s (± 0.90%) 65.20s (± 0.92%) ~ 64.19s 65.98s p=0.230 n=6
Emit Time 0.15s 0.15s ~ ~ ~ p=1.000 n=6
Total Time 71.87s (± 0.86%) 72.36s (± 0.86%) ~ 71.35s 73.15s p=0.230 n=6
self-build-src - bun (v1.1.17, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,260 1,225,260 ~ ~ ~ p=1.000 n=6
Types 261,341 261,341 ~ ~ ~ p=1.000 n=6
Memory used 9,024,980k (± 0.05%) 8,986,079k (± 0.37%) -38,901k (- 0.43%) 8,922,529k 9,011,426k p=0.005 n=6
Parse Time 4.06s (± 0.71%) 4.12s (± 0.99%) +0.06s (+ 1.60%) 4.05s 4.16s p=0.037 n=6
Bind Time 1.71s (± 1.17%) 1.73s (± 0.70%) +0.02s (+ 1.36%) 1.72s 1.75s p=0.041 n=6
Check Time 31.67s (± 0.17%) 30.91s (± 4.16%) ~ 29.24s 31.87s p=0.936 n=6
Emit Time 1.76s (± 0.84%) 1.75s (± 4.54%) ~ 1.61s 1.84s p=1.000 n=6
Total Time 39.22s (± 0.16%) 38.53s (± 3.50%) ~ 36.79s 39.53s p=0.378 n=6
self-build-src-public-api - bun (v1.1.17, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,260 1,225,260 ~ ~ ~ p=1.000 n=6
Types 261,341 261,341 ~ ~ ~ p=1.000 n=6
Memory used 4,383,876k (± 3.45%) 5,556,996k (±34.14%) ~ 4,417,514k 9,025,179k p=0.689 n=6
Parse Time 3.90s (± 0.88%) 3.91s (± 1.28%) ~ 3.87s 4.00s p=1.000 n=6
Bind Time 1.82s (± 1.73%) 1.91s (± 2.98%) 🔻+0.09s (+ 5.05%) 1.85s 2.00s p=0.008 n=6
Check Time 33.57s (± 2.08%) 33.06s (± 1.99%) ~ 31.91s 33.61s p=0.810 n=6
Emit Time 1.75s (± 3.37%) 1.70s (± 6.15%) ~ 1.49s 1.77s p=0.471 n=6
Total Time 41.06s (± 1.78%) 40.60s (± 1.69%) ~ 39.46s 41.14s p=1.000 n=6
self-compiler - bun (v1.1.17, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 258,194 258,194 ~ ~ ~ p=1.000 n=6
Types 104,737 104,737 ~ ~ ~ p=1.000 n=6
Memory used 422,498k (± 1.07%) 414,716k (± 1.26%) -7,783k (- 1.84%) 411,644k 425,304k p=0.031 n=6
Parse Time 3.28s (± 1.52%) 3.28s (± 0.60%) ~ 3.26s 3.31s p=0.936 n=6
Bind Time 1.65s (± 1.52%) 1.66s (± 2.07%) ~ 1.60s 1.70s p=0.223 n=6
Check Time 20.63s (± 0.49%) 20.77s (± 0.48%) ~ 20.66s 20.94s p=0.065 n=6
Emit Time 1.26s (± 2.29%) 1.25s (± 3.59%) ~ 1.19s 1.30s p=0.748 n=6
Total Time 26.81s (± 0.42%) 26.96s (± 0.48%) ~ 26.81s 27.17s p=0.092 n=6
ts-pre-modules - bun (v1.1.17, x64)
Errors 35 35 ~ ~ ~ p=1.000 n=6
Symbols 224,565 224,565 ~ ~ ~ p=1.000 n=6
Types 93,734 93,734 ~ ~ ~ p=1.000 n=6
Memory used 355,399k (± 0.35%) 356,775k (± 0.26%) +1,376k (+ 0.39%) 355,710k 358,199k p=0.045 n=6
Parse Time 2.31s (± 0.59%) 2.34s (± 1.44%) ~ 2.31s 2.40s p=0.183 n=6
Bind Time 1.44s (± 0.95%) 1.43s (± 1.20%) ~ 1.41s 1.45s p=0.282 n=6
Check Time 14.35s (± 0.39%) 14.52s (± 0.45%) +0.16s (+ 1.14%) 14.44s 14.61s p=0.005 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 18.11s (± 0.33%) 18.29s (± 0.35%) +0.18s (+ 0.98%) 18.22s 18.41s p=0.005 n=6
vscode - bun (v1.1.17, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,879,835 2,879,835 ~ ~ ~ p=1.000 n=6
Types 975,372 975,372 ~ ~ ~ p=1.000 n=6
Memory used 3,105,899k (± 0.27%) 3,137,837k (± 0.15%) +31,938k (+ 1.03%) 3,132,157k 3,143,317k p=0.005 n=6
Parse Time 13.67s (± 0.37%) 13.74s (± 0.52%) +0.07s (+ 0.52%) 13.64s 13.86s p=0.037 n=6
Bind Time 4.64s (± 1.26%) 4.56s (± 0.71%) -0.08s (- 1.76%) 4.50s 4.59s p=0.005 n=6
Check Time 79.24s (± 0.74%) 79.48s (± 0.24%) ~ 79.23s 79.74s p=0.092 n=6
Emit Time 24.14s (± 0.89%) 23.81s (± 0.43%) -0.32s (- 1.34%) 23.70s 23.97s p=0.008 n=6
Total Time 121.69s (± 0.53%) 121.59s (± 0.20%) ~ 121.25s 121.85s p=0.689 n=6
webpack - bun (v1.1.17, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 267,117 267,117 ~ ~ ~ p=1.000 n=6
Types 108,775 108,775 ~ ~ ~ p=1.000 n=6
Memory used 429,430k (± 0.17%) 425,739k (± 0.46%) -3,691k (- 0.86%) 422,664k 428,124k p=0.005 n=6
Parse Time 3.10s (± 0.34%) 3.10s (± 1.02%) ~ 3.06s 3.15s p=0.625 n=6
Bind Time 1.38s (± 1.25%) 1.42s (± 2.56%) +0.04s (+ 2.66%) 1.38s 1.48s p=0.043 n=6
Check Time 13.84s (± 0.67%) 13.77s (± 0.59%) ~ 13.66s 13.84s p=0.470 n=6
Emit Time 0.00s 0.01s (±154.76%) ~ 0.00s 0.02s p=0.174 n=6
Total Time 18.32s (± 0.43%) 18.28s (± 0.34%) ~ 18.17s 18.34s p=0.936 n=6
xstate-main - bun (v1.1.17, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 525,251 525,251 ~ ~ ~ p=1.000 n=6
Types 178,574 178,574 ~ ~ ~ p=1.000 n=6
Memory used 463,436k (± 0.42%) 458,406k (± 0.11%) -5,029k (- 1.09%) 458,071k 459,306k p=0.005 n=6
Parse Time 2.73s (± 0.91%) 2.70s (± 0.51%) -0.04s (- 1.34%) 2.68s 2.71s p=0.007 n=6
Bind Time 1.02s (± 6.60%) 1.07s (± 2.20%) ~ 1.04s 1.10s p=0.191 n=6
Check Time 15.85s (± 0.46%) 15.88s (± 0.23%) ~ 15.83s 15.94s p=0.296 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 19.61s (± 0.47%) 19.64s (± 0.25%) ~ 19.59s 19.71s p=0.335 n=6
System info unknown
Hosts
  • bun (v1.1.17, x64)
Scenarios
  • Compiler-Unions - bun (v1.1.17, x64)
  • angular-1 - bun (v1.1.17, x64)
  • mui-docs - bun (v1.1.17, x64)
  • self-build-src - bun (v1.1.17, x64)
  • self-build-src-public-api - bun (v1.1.17, x64)
  • self-compiler - bun (v1.1.17, x64)
  • ts-pre-modules - bun (v1.1.17, x64)
  • vscode - bun (v1.1.17, x64)
  • webpack - bun (v1.1.17, x64)
  • xstate-main - bun (v1.1.17, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-Unions - node (v18.15.0, x64)
Errors 30 30 ~ ~ ~ p=1.000 n=6
Symbols 62,153 62,153 ~ ~ ~ p=1.000 n=6
Types 50,242 50,242 ~ ~ ~ p=1.000 n=6
Memory used 192,831k (± 0.76%) 192,968k (± 0.88%) ~ 191,649k 195,158k p=0.230 n=6
Parse Time 1.57s (± 1.24%) 1.58s (± 0.48%) ~ 1.57s 1.59s p=0.383 n=6
Bind Time 0.85s (± 1.05%) 0.85s (± 0.89%) ~ 0.84s 0.86s p=0.798 n=6
Check Time 11.16s (± 0.42%) 11.06s (± 0.25%) -0.10s (- 0.87%) 11.05s 11.12s p=0.009 n=6
Emit Time 3.29s (± 0.85%) 3.16s (± 0.49%) 🟩-0.13s (- 3.85%) 3.13s 3.17s p=0.005 n=6
Total Time 16.87s (± 0.25%) 16.65s (± 0.20%) -0.22s (- 1.29%) 16.61s 16.70s p=0.005 n=6
angular-1 - node (v18.15.0, x64)
Errors 5 5 ~ ~ ~ p=1.000 n=6
Symbols 944,114 944,114 ~ ~ ~ p=1.000 n=6
Types 407,050 407,050 ~ ~ ~ p=1.000 n=6
Memory used 1,218,358k (± 0.00%) 1,217,900k (± 0.00%) -458k (- 0.04%) 1,217,836k 1,217,960k p=0.005 n=6
Parse Time 7.95s (± 0.56%) 7.92s (± 0.46%) ~ 7.88s 7.97s p=0.170 n=6
Bind Time 2.23s (± 0.79%) 2.25s (± 0.59%) ~ 2.24s 2.27s p=0.241 n=6
Check Time 35.77s (± 0.45%) 35.66s (± 0.37%) ~ 35.55s 35.90s p=0.298 n=6
Emit Time 16.15s (± 0.31%) 15.93s (± 0.89%) -0.22s (- 1.34%) 15.68s 16.07s p=0.006 n=6
Total Time 62.10s (± 0.33%) 61.76s (± 0.25%) -0.34s (- 0.55%) 61.52s 61.99s p=0.020 n=6
mui-docs - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,133,039 2,133,039 ~ ~ ~ p=1.000 n=6
Types 926,444 926,444 ~ ~ ~ p=1.000 n=6
Memory used 2,115,376k (± 0.01%) 2,115,228k (± 0.00%) -148k (- 0.01%) 2,115,209k 2,115,240k p=0.005 n=6
Parse Time 6.65s (± 0.11%) 6.62s (± 0.15%) -0.03s (- 0.40%) 6.61s 6.63s p=0.004 n=6
Bind Time 2.33s (± 0.90%) 2.30s (± 0.22%) -0.03s (- 1.14%) 2.30s 2.31s p=0.036 n=6
Check Time 70.61s (± 0.58%) 71.01s (± 0.30%) ~ 70.63s 71.20s p=0.128 n=6
Emit Time 0.14s (± 3.77%) 0.14s (± 4.05%) ~ 0.13s 0.14s p=0.640 n=6
Total Time 79.73s (± 0.52%) 80.07s (± 0.27%) ~ 79.69s 80.27s p=0.128 n=6
self-build-src - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,260 1,225,260 ~ ~ ~ p=1.000 n=6
Types 261,341 261,341 ~ ~ ~ p=1.000 n=6
Memory used 2,341,363k (± 0.06%) 2,336,365k (± 0.05%) -4,998k (- 0.21%) 2,334,569k 2,337,630k p=0.005 n=6
Parse Time 6.05s (± 1.26%) 6.02s (± 1.37%) ~ 5.91s 6.14s p=0.470 n=6
Bind Time 2.25s (± 0.81%) 2.26s (± 0.54%) ~ 2.24s 2.27s p=0.572 n=6
Check Time 40.24s (± 0.58%) 39.97s (± 0.29%) -0.27s (- 0.68%) 39.79s 40.12s p=0.045 n=6
Emit Time 3.13s (± 5.82%) 3.05s (± 2.39%) ~ 2.99s 3.18s p=0.471 n=6
Total Time 51.67s (± 0.71%) 51.30s (± 0.38%) ~ 50.98s 51.59s p=0.093 n=6
self-build-src-public-api - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,260 1,225,260 ~ ~ ~ p=1.000 n=6
Types 261,341 261,341 ~ ~ ~ p=1.000 n=6
Memory used 2,414,683k (± 0.03%) 2,414,002k (± 0.06%) ~ 2,412,449k 2,415,824k p=0.471 n=6
Parse Time 6.26s (± 0.79%) 6.28s (± 1.44%) ~ 6.15s 6.35s p=0.471 n=6
Bind Time 2.01s (± 1.13%) 2.24s (± 4.48%) 🔻+0.23s (+11.44%) 2.04s 2.31s p=0.005 n=6
Check Time 40.72s (± 0.29%) 40.11s (± 0.46%) -0.61s (- 1.49%) 39.91s 40.45s p=0.005 n=6
Emit Time 3.19s (± 3.15%) 3.17s (± 4.13%) ~ 3.05s 3.34s p=0.748 n=6
Total Time 52.18s (± 0.27%) 51.80s (± 0.48%) -0.38s (- 0.72%) 51.51s 52.11s p=0.020 n=6
self-compiler - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 258,194 258,194 ~ ~ ~ p=1.000 n=6
Types 104,737 104,737 ~ ~ ~ p=1.000 n=6
Memory used 427,458k (± 0.00%) 427,007k (± 0.01%) -451k (- 0.11%) 426,970k 427,035k p=0.005 n=6
Parse Time 4.08s (± 0.75%) 4.05s (± 0.36%) -0.04s (- 0.86%) 4.03s 4.07s p=0.042 n=6
Bind Time 1.61s (± 0.85%) 1.62s (± 1.00%) ~ 1.60s 1.64s p=0.328 n=6
Check Time 22.01s (± 0.51%) 21.88s (± 0.47%) ~ 21.73s 22.03s p=0.066 n=6
Emit Time 1.54s (± 0.80%) 1.60s (± 2.22%) 🔻+0.07s (+ 4.56%) 1.57s 1.66s p=0.005 n=6
Total Time 29.24s (± 0.41%) 29.15s (± 0.22%) ~ 29.08s 29.26s p=0.170 n=6
ts-pre-modules - node (v18.15.0, x64)
Errors 35 35 ~ ~ ~ p=1.000 n=6
Symbols 224,565 224,565 ~ ~ ~ p=1.000 n=6
Types 93,734 93,734 ~ ~ ~ p=1.000 n=6
Memory used 369,455k (± 0.01%) 369,013k (± 0.01%) -442k (- 0.12%) 368,953k 369,089k p=0.005 n=6
Parse Time 2.76s (± 0.70%) 2.75s (± 0.84%) ~ 2.71s 2.78s p=0.462 n=6
Bind Time 1.58s (± 0.40%) 1.59s (± 0.83%) ~ 1.57s 1.61s p=0.081 n=6
Check Time 15.45s (± 0.50%) 15.27s (± 0.14%) -0.18s (- 1.18%) 15.24s 15.30s p=0.005 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 19.79s (± 0.40%) 19.61s (± 0.12%) -0.18s (- 0.90%) 19.57s 19.64s p=0.005 n=6
vscode - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,879,835 2,879,835 ~ ~ ~ p=1.000 n=6
Types 975,372 975,372 ~ ~ ~ p=1.000 n=6
Memory used 3,042,553k (± 0.00%) 3,042,051k (± 0.00%) -502k (- 0.02%) 3,041,973k 3,042,131k p=0.005 n=6
Parse Time 13.63s (± 0.66%) 13.59s (± 0.28%) ~ 13.54s 13.64s p=0.747 n=6
Bind Time 4.20s (± 2.06%) 4.20s (± 0.48%) ~ 4.17s 4.23s p=0.102 n=6
Check Time 74.31s (± 2.29%) 74.18s (± 2.58%) ~ 72.59s 76.83s p=0.810 n=6
Emit Time 22.65s (± 6.40%) 22.39s (± 8.36%) ~ 20.12s 24.24s p=0.810 n=6
Total Time 114.79s (± 0.28%) 114.37s (± 0.29%) ~ 113.97s 114.81s p=0.066 n=6
webpack - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 267,117 267,117 ~ ~ ~ p=1.000 n=6
Types 108,775 108,775 ~ ~ ~ p=1.000 n=6
Memory used 411,571k (± 0.01%) 411,215k (± 0.02%) -356k (- 0.09%) 411,140k 411,334k p=0.005 n=6
Parse Time 3.83s (± 0.46%) 3.80s (± 0.61%) ~ 3.77s 3.83s p=0.076 n=6
Bind Time 1.70s (± 0.44%) 1.68s (± 0.53%) -0.02s (- 1.08%) 1.67s 1.69s p=0.011 n=6
Check Time 16.73s (± 0.45%) 16.84s (± 0.37%) +0.11s (+ 0.67%) 16.75s 16.94s p=0.025 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 22.25s (± 0.34%) 22.32s (± 0.26%) ~ 22.24s 22.42s p=0.092 n=6
xstate-main - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 525,251 525,251 ~ ~ ~ p=1.000 n=6
Types 178,574 178,574 ~ ~ ~ p=1.000 n=6
Memory used 462,869k (± 0.07%) 462,639k (± 0.03%) ~ 462,522k 462,869k p=0.128 n=6
Parse Time 2.65s (± 0.52%) 2.66s (± 0.31%) ~ 2.65s 2.67s p=0.094 n=6
Bind Time 0.98s (± 0.42%) 0.99s (± 0.41%) +0.01s (+ 0.68%) 0.98s 0.99s p=0.034 n=6
Check Time 15.15s (± 0.26%) 15.10s (± 0.44%) ~ 15.00s 15.20s p=0.170 n=6
Emit Time 0.00s 0.00s (±244.70%) ~ 0.00s 0.02s p=0.405 n=6
Total Time 18.77s (± 0.23%) 18.75s (± 0.39%) ~ 18.64s 18.86s p=0.470 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Compiler-Unions - node (v18.15.0, x64)
  • angular-1 - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • ts-pre-modules - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate-main - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 3,425ms (± 0.52%) 3,425ms (± 0.31%) ~ 3,411ms 3,437ms p=0.630 n=6
Req 2 - geterr 7,551ms (± 0.72%) 7,245ms (± 0.61%) 🟩-306ms (- 4.05%) 7,205ms 7,318ms p=0.005 n=6
Req 3 - references 394ms (± 0.38%) 403ms (± 0.22%) +9ms (+ 2.20%) 402ms 404ms p=0.004 n=6
Req 4 - navto 341ms (± 0.19%) 338ms (± 0.66%) -3ms (- 0.83%) 335ms 342ms p=0.046 n=6
Req 5 - completionInfo count 1,357 1,357 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 130ms (± 1.19%) 117ms (± 0.47%) 🟩-14ms (-10.38%) 116ms 117ms p=0.004 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,420ms (± 0.51%) 2,413ms (± 0.61%) ~ 2,396ms 2,439ms p=0.574 n=6
Req 2 - geterr 3,798ms (± 0.33%) 3,729ms (± 0.53%) -69ms (- 1.81%) 3,707ms 3,758ms p=0.005 n=6
Req 3 - references 277ms (± 0.42%) 283ms (± 0.37%) +6ms (+ 2.11%) 281ms 284ms p=0.005 n=6
Req 4 - navto 226ms (± 0.36%) 226ms (± 0.43%) ~ 225ms 227ms p=0.394 n=6
Req 5 - completionInfo count 1,519 1,519 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 73ms (± 8.98%) 77ms (± 6.65%) ~ 73ms 84ms p=0.195 n=6
xstate-main-1-tsserver - node (v18.15.0, x64)
Req 1 - updateOpen 6,472ms (± 5.99%) 6,237ms (± 0.47%) ~ 6,219ms 6,297ms p=0.470 n=6
Req 2 - geterr 1,632ms (± 7.37%) 1,626ms (± 8.19%) ~ 1,354ms 1,692ms p=1.000 n=6
Req 3 - references 109ms (± 9.13%) 104ms (± 0.86%) ~ 103ms 105ms p=0.062 n=6
Req 4 - navto 579ms (± 7.13%) 594ms (± 2.58%) ~ 566ms 606ms p=0.574 n=6
Req 5 - completionInfo count 3,413 3,413 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 1,248ms (± 1.48%) 1,249ms (± 1.25%) ~ 1,232ms 1,276ms p=0.810 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstate-main-1-tsserver - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 189.19ms (± 0.16%) 186.70ms (± 0.20%) -2.49ms (- 1.32%) 185.34ms 193.79ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 283.68ms (± 0.27%) 281.96ms (± 0.28%) -1.72ms (- 0.61%) 274.83ms 290.72ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 277.05ms (± 0.27%) 275.43ms (± 0.27%) -1.62ms (- 0.58%) 268.60ms 278.71ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 276.32ms (± 0.28%) 274.82ms (± 0.28%) -1.50ms (- 0.54%) 268.22ms 278.28ms p=0.000 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey jakebailey closed this Jun 30, 2024
@kurtextrem
Copy link

For the curious why this has been closed, see Jake's reply on X: https://x.com/andhaveaniceday/status/1775243613506867262?t=dzeDZE6NiL6MgjodaKIUIA&s=19

@jakebailey
Copy link
Member Author

Well, mainly that we manually var'd everything critical and anything else seems to just be noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.