Supports incremental build in tsc --b --w mode #29161

sheetalkamat · 2018-12-26T20:33:56Z

Some notes:

Since we use builder, the first time program is created, its going to emit all the files and not just changed - just like it does in tsc --watch mode
We do not cache source files yet and that's something to be looked at later.
Two things can be done:
- To keep source files for program thats being updated in last few cycles
- To release the source files if size exceeds or program is not used for few build cycles
- Adding this to Create tsbuild public API and use incremental build in tsbuild --watch mode #28640
We use hash from source file's text as the version.
Because in tsbuild we get semantic diagnostics before emit, the EmitAndSemanticDiagnosticsBuilderProgram's existing assumption was not satisfied (that is changed files are already emitted before getting semantic diagnostics). So in such case, we store list of files to emit later so as to help with that case.
In this we also have a new Builder method to release the program and its source files to ensure that we aren't holding on program after emit and/or semantic diagnostics are complete. That way the builder state can be in memory for comparison and finding out what are new changes that need emitting while creating new program but we aren't holding onto these costly (memory wise) resources. (all the program methods are thus available only till program is released.

Here are the perf numbers:
Local change::

File change detected. Starting incremental compilation...

Project 'SubProject 4' is out of date because oldest output 'someOutputFile.js' is older than newest input 'SubProject 2'

Building project 'SubProject 4'...

                                Updating unchanged output timestamps of project 'SubProject 4'...

Before                          After
Files:                  8305    Files:                  8305
Lines:                509633    Lines:                509633
Nodes:               1741390    Nodes:               1741390
Identifiers:          563590    Identifiers:          563590
Symbols:              597260    Symbols:              316035
Types:                175190    Types:                   232
Memory used:        1418992K    Memory used:        1859282K
I/O Read time:         2.83s    I/O Read time:         3.85s
Parse time:            2.43s    Parse time:            3.42s
Program time:         23.45s    Program time:         33.07s
Bind time:             1.56s    Bind time:             1.71s
Check time:           11.86s    Check time:            0.04s
transformTime time:   13.72s    transformTime time:    0.25s
Source Map time:       2.58s    Source Map time:       0.01s
commentTime time:      1.46s    commentTime time:      0.00s
printTime time:       27.08s    printTime time:        0.28s
Emit time:            27.08s    Emit time:             0.28s
I/O Write time:       16.16s    I/O Write time:        0.04s
Total time:           63.96s    Total time:           35.09s
Building project 'SubProject 5'...

                                Updating unchanged output timestamps of project 'SubProject 5'...

Before                          After
Files:                  1875    Files:            1875
Lines:                207731    Lines:          207731
Nodes:                632955    Nodes:          632955
Identifiers:          212573    Identifiers:    212573
Symbols:              162334    Symbols:        125245
Types:                 22975    Types:              71
Memory used:        1095817K    Memory used:   311574K
I/O Read time:         1.13s    I/O Read time:   0.74s
Parse time:            2.03s    Parse time:      0.87s
Program time:          9.64s    Program time:    5.97s
Bind time:             0.67s    Bind time:       0.48s
Check time:            2.17s
transformTime time:    1.22s
Source Map time:       0.77s
commentTime time:      0.34s
printTime time:        4.16s
Emit time:             4.16s
I/O Write time:        1.92s
Total time:           16.64s    Total time:      6.45s
Building project 'SubProject 6'...

                                Updating unchanged output timestamps of project 'SubProject 6'...

Before                          After
Files:                  2396    Files:            2396
Lines:                221677    Lines:          221677
Nodes:                669898    Nodes:          669898
Identifiers:          221017    Identifiers:    221017
Symbols:              189874    Symbols:        143939
Types:                 30412    Types:              71
Memory used:        1442324K    Memory used:   333909K
I/O Read time:         0.92s    I/O Read time:   0.91s
Parse time:            1.40s    Parse time:      1.17s
Program time:          8.91s    Program time:    7.73s
Bind time:             0.43s    Bind time:       0.39s
Check time:            1.74s
transformTime time:    0.67s
Source Map time:       0.18s
commentTime time:      0.10s
printTime time:        1.67s
Emit time:             1.67s
I/O Write time:        1.40s
Total time:           12.74s    Total time:      8.12s
Building project 'SubProject 7'...

                                Updating unchanged output timestamps of project 'SubProject 7'...

Before                          After
Files:                  9326    Files:            9326
Lines:                552828    Lines:          552830
Nodes:               1896877    Nodes:         1896879
Identifiers:          605603    Identifiers:    605605
Symbols:              608250    Symbols:        330625
Types:                173986    Types:              71
Memory used:        2367385K    Memory used:   953536K
I/O Read time:         4.22s    I/O Read time:   3.84s
Parse time:            3.18s    Parse time:      2.02s
Program time:         36.41s    Program time:   27.79s
Bind time:             2.73s    Bind time:       1.19s
Check time:           13.15s
transformTime time:   15.36s
Source Map time:       2.35s
commentTime time:      1.52s
printTime time:       28.95s
Emit time:            28.95s
I/O Write time:       13.41s
Total time:           81.23s    Total time:     28.98s
Building project 'SubProject 8'...

                                Updating unchanged output timestamps of project 'SubProject 8'...

Before                          After
Files:                  6146    Files:             6146
Lines:                367180    Lines:           367182
Nodes:               1201703    Nodes:          1201705
Identifiers:          386452    Identifiers:     386454
Symbols:              406182    Symbols:         232990
Types:                 94077    Types:               71
Memory used:        1704251K    Memory used:   1333457K
I/O Read time:         3.06s    I/O Read time:    2.50s
Parse time:            3.61s    Parse time:       2.06s
Program time:         31.86s    Program time:    19.05s
Bind time:             1.76s    Bind time:        0.77s
Check time:           10.90s
transformTime time:    7.64s
Source Map time:       0.96s
commentTime time:      0.91s
printTime time:       13.81s
Emit time:            13.81s
I/O Write time:        6.67s
Total time:           58.33s    Total time:      19.82s
Building project 'SubProject 9'...

                                Updating unchanged output timestamps of project 'SubProject 9'...

Before                          After
Files:                  3781    Files:             3781
Lines:                264899    Lines:           264899
Nodes:                834360    Nodes:           834360
Identifiers:          278052    Identifiers:     278052
Symbols:              261052    Symbols:         178924
Types:                 52797    Types:               71
Memory used:        2175398K    Memory used:   1620896K
I/O Read time:         1.67s    I/O Read time:    1.51s
Parse time:            1.98s    Parse time:       1.19s
Program time:         16.03s    Program time:    11.75s
Bind time:             0.77s    Bind time:        0.49s
Check time:            4.86s
transformTime time:    4.24s
Source Map time:       0.40s
commentTime time:      0.21s
printTime time:        6.63s
Emit time:             6.63s
I/O Write time:        2.03s
Total time:           28.29s    Total time:      12.24s


Solution Total Time:
Before: 261.19, After: 110.7

Local change:

File change detected. Starting incremental compilation...

Project 'SubProject 4' is out of date because oldest output 'someOutputFile.js' is older than newest input 'SubProject 2'

Building project 'SubProject 4'...

                                Updating unchanged output timestamps of project 'SubProject 4'...

Before                          After
Files:                  8305    Files:                 8305
Lines:                509635    Lines:               509635
Nodes:               1741406    Nodes:              1741406
Identifiers:          563597    Identifiers:         563597
Symbols:              597262    Symbols:             315957
Types:                175191    Types:                  114
Memory used:        3243603K    Memory used:        690192K
I/O Read time:         3.23s    I/O Read time:        3.28s
Parse time:            4.00s    Parse time:           2.31s
Program time:         35.91s    Program time:        24.59s
Bind time:             1.75s    Bind time:            1.27s
Check time:           15.76s    Check time:           0.01s
transformTime time:   15.10s    transformTime time:   0.00s
Source Map time:       2.93s    Source Map time:      0.00s
commentTime time:      1.85s    commentTime time:     0.00s
printTime time:       30.68s    printTime time:       0.00s
Emit time:            30.68s    Emit time:            0.00s
I/O Write time:       14.47s    I/O Write time:       0.00s
Total time:           84.10s    Total time:          25.87s
                                Project 'SubProject 5' is up to date with .d.ts files from its dependencies

                                Updating output timestamps of project 'SubProject 5'...

                                Project 'SubProject 6' is up to date with .d.ts files from its dependencies

                                Updating output timestamps of project 'SubProject 6'...

                                Project 'SubProject 7' is up to date with .d.ts files from its dependencies

                                Updating output timestamps of project 'SubProject 7'...

                                Project 'SubProject 8' is up to date with .d.ts files from its dependencies

                                Updating output timestamps of project 'SubProject 8'...

                                Project 'SubProject 9' is up to date with .d.ts files from its dependencies

                                Updating output timestamps of project 'SubProject 9'...


Solution Total Time:
Before: 84.1, After: 25.87
Found 0 errors. Watching for file changes.

…ide detailed information on watch invokations

…ncremental builds

…on to set Program in state to undefined for storing.

This helps in storing the semantic diagnostics

… of incremental build This ensures that after `tsbuild` after incremental build of `tsbuild -w` doesnt result in unnecessary rebuilds

…oject reference with prepend since it cant tell changes in js/map files

…dling These will fail since its still TODO

…state This helps us revert to state where we pretend as if emit is not done (since we do not do emit if there are errors)

src/compiler/program.ts

sheetalkamat · 2019-01-10T17:30:22Z

Ping @sandersn @rbuckton @weswigham @DanielRosenwasser Would like to get this in soon.

src/compiler/core.ts

src/compiler/sys.ts

src/compiler/builder.ts

src/compiler/tsbuild.ts

src/compiler/watch.ts

sandersn

I looked over everything but I need to learn more about watch mode + tsbuild in person before signing off.

src/compiler/core.ts

src/compiler/tsbuild.ts

src/compiler/builder.ts

src/compiler/watch.ts

src/compiler/program.ts

src/compiler/builder.ts

sheetalkamat · 2019-01-17T20:52:50Z

@sandersn comments?

xaviergonz · 2019-01-17T23:06:03Z

Just wondering, wouldn't it be even faster if the cache was also based on file size as first check before doing the actual hash?

sheetalkamat added 11 commits December 21, 2018 12:53

Use watch factory instead of direct host functions in tsbuild to prov…

0cabb00

…ide detailed information on watch invokations

Make BuilderProgram as Program

9e05abc

Make SolutionBuilder handle BuilderProgram in preparation to handle i…

48baa42

…ncremental builds

Revert BuilderProgram to be redirected object to Program in preparati…

56a76d8

…on to set Program in state to undefined for storing.

Add method to release held Program in BuilderProgram

69193d9

Use oldProgram to create the new Program.

47f5106

This helps in storing the semantic diagnostics

Use emit builder to emit only changed files.

f1949bb

Update the timestamps of outputs that dont need to be written because…

7b290fd

… of incremental build This ensures that after `tsbuild` after incremental build of `tsbuild -w` doesnt result in unnecessary rebuilds

Handle prepend in incremental build. Always emit when program uses pr…

0d9038c

…oject reference with prepend since it cant tell changes in js/map files

Write the tests for incremental build and declaration emit errors han…

b360ff7

…dling These will fail since its still TODO

Handle declaration emit errors in tsbuild mode by backing up builder …

69abc12

…state This helps us revert to state where we pretend as if emit is not done (since we do not do emit if there are errors)

sheetalkamat mentioned this pull request Dec 26, 2018

Create tsbuild public API and use incremental build in tsbuild --watch mode #28640

Closed

9 tasks

sheetalkamat requested review from weswigham, DanielRosenwasser, rbuckton and sandersn December 26, 2018 20:55

ajafff reviewed Dec 26, 2018

View reviewed changes

src/compiler/program.ts Show resolved Hide resolved

sheetalkamat added 2 commits December 27, 2018 10:36

Use DirectoryStructureHost for fileExists and readFile

42484b5

Merge branch 'master' into incrementalBuild

1b37830

sandersn reviewed Jan 10, 2019

View reviewed changes

src/compiler/core.ts Outdated Show resolved Hide resolved

Fix typo

abc8618

sandersn reviewed Jan 10, 2019

View reviewed changes

src/compiler/sys.ts Show resolved Hide resolved

Merge branch 'master' into incrementalBuild

c1edbb8

sandersn self-assigned this Jan 14, 2019