Calculate a partial fence difference based on git changes since a provided oid #101

Adjective-Object · 2021-08-11T22:32:30Z

Adds a CLI flag to only check fences changed since a given git OID / refname.
This diffs against the current git index to serve everything from git's object DB instead of going to disk.

The process is:

user specifies --sinceGitHash on the command line
we get the diff with the index
from the diff we read the changes to the fence.json files, and diff the current vs old versions of fences to create fenceDiffs
from the diff we parse the old and new versions of the changed typescript files to produce import sets, then diff those importSets to figure out the added/removeed import specifiers in each script.
We use those diffs to check which script files to re-evaluate
- when a script file gains a new import, we re-check it against all fences that apply to it. If that import is to an internal file, each of its imports will also be checked
- when a fence loses an "imports" rule, we re-check all files in the scope of that fence
- when a fence loses an "dependencies" rule, we re-check all files in the scope of that fence
- if a fence loses an "exports" rule, we abort, since you have to check all fences in that case (we can't know what is importing the fence)

Adds a faster file provider that discovers source and fence files with a fast asyncronous file walk rather than by relying on typescript to perform a syncronous walk + parse of the program. We still use typescript to parse compilerOptions, but since we no longer rely on the typescript program walk to identify files, we stub out file discovery so only the initial config partse happens. Also adds config options to support switching between the two providers, since it will only work when all source files you intend to check fences against fall under your rootDirs. This is the case for OWA, but I don't know what other consumers look like. Depends on u/mahuangh/faster-source-providers-2

…set files in fdir source file provider

Adjective-Object · 2021-08-11T22:33:47Z

I realized while reviewing this change that the current way this is structured mishandles the adding of empty "dependencies" or "imports" sections. It treats that as a noop when in reality is a more constraining fence.

Adjective-Object · 2021-08-12T20:50:16Z

Added tests to the fence and import diffing logic, and covered the cases where sections are added / removed. Also updated the diffing logic to consider the removal of tags, so that when a tag is removed the whole fence is re-checked.

Together these cases should handle some false-negatives from the incremental fence check that we were missing in the full fence check.

…roviders-3

…-source-providers-6-partial-evaluation

Adjective-Object

This change got pretty big after adding tests for the utility functions. Let me know if you want me to break it up into pieces again.

src/core/runner.ts

.vscode/launch.json

src/core/cli.ts

src/core/runner.ts

src/types/config/Config.ts

src/utils/getFenceAndImportDiffsFromGit.ts

src/utils/getFenceDiff.ts

src/utils/getOptions.ts

src/utils/getPartialCheckFromImportDiffs.ts

smikula

Lots of comments. This is a tricky change — nice work! It's frustrating that there are so many cases where we need to bail out on validation, but it should be a big win for the most common cases.

…imports

… tag removal

Adjective-Object · 2021-09-01T20:01:52Z

I think I updated to the latest feedback, let me know if I missed something

src/utils/diffing/getPartialCheckFromImportDiffs.ts

Maxwell Huang-Hobbs and others added 10 commits August 11, 2021 16:52

progress -> progressBar

ddfefb1

progress -> progressBar

6bb301a

Merge branch 'master' into u/mahuangh/faster-source-providers-3

525e910

support definition files, simplify resolveImportsFromFile, exclude as…

cce66e2

…set files in fdir source file provider

getScriptFileExtensions

ac076f2

add partial evaluation based on git checkout

b012df6

remove FdirSourceFileProvider's dependencies

6e181a2

remove looseRootFileDiscovery

24a4fe6

remove extra diff (fenceJobs -> normalizedFiles

c937ede

Maxwell Huang-Hobbs added 6 commits August 11, 2021 18:35

add json to the moduleFileExtensions in jest

155cf8f

Added tests for getPartialCheck / getFenceDiff

b90c25e

add launch.json & debug current test

e9b7322

Prefer preProcessFile in getTsImportSetFromSourceString

ff344ab

consider tag diffs during partial evaluation

8ba8b2a

remove commented code from getTsImportSetFromSourceString

bae49ea

Adjective-Object mentioned this pull request Aug 13, 2021

Add a file provider that avoids ts program walk #100

Merged

Maxwell Huang-Hobbs added 5 commits August 13, 2021 12:02

update to new getScriptFIleExtensions from u/mahuangh/faster-source-p…

bdb73d7

…roviders-3

Merge remote-tracking branch 'upstream/master' into u/mahuangh/faster…

e7a2629

…-source-providers-6-partial-evaluation

get diffcount of index properly

db855d8

update log messages to replace -- with new sentences

ce3af4a

remove errant \n in fenceless warnings

8ad6d6d

Adjective-Object commented Aug 17, 2021

View reviewed changes

src/core/runner.ts Outdated Show resolved Hide resolved