Skip to content

Include type reference directives in symlink cache, wait until program is present to create it #44259

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

andrewbranch
Copy link
Member

Fixes a half-regression half-new-issue noted in #40584 (comment) where (some) auto imports and fixes inside node_modules/.pnpm were showing up again. The bug was surfaced by #43149, but only through a coincidental change in order of operations. The real root causes were:

  1. The symlinks discovered by automatic type reference directives were not being added to the symlink cache.
  2. The host Project was creating an empty symlink cache too early—by deferring until after program creation, it can seed the cache with the symlinks discovered during module resolution for the files in that program.

While testing, I noticed that it was still possible to get import statement completions with .pnpm module specifiers for transitive dependencies, which are not symlinked into your own node_modules. When I first wrote the .pnpm exclusion stuff, I allowed the .pnpm realpath to be used as a last resort if no suitable symlinks were found, primarily to avoid breaking declaration emit, which (naturally) assumes that you can always name a relative path from one module to another. I now allow the module specifier resolution host used by the services layer to opt out of getting these last-resort paths, so it can filter these suggestions out of its auto-imports.

@typescript-bot typescript-bot added Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels May 25, 2021
@andrewbranch andrewbranch requested a review from sheetalkamat May 25, 2021 23:54
@andrewbranch
Copy link
Member Author

@typescript-bot pack this

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 26, 2021

Heya @andrewbranch, I've started to run the tarball bundle task on this PR at 3794e04. You can monitor the build here.

@Jack-Works
Copy link
Contributor

It seems like the packing has failed

@Jack-Works
Copy link
Contributor

friendly ping @andrewbranch it failed to build

@andrewbranch
Copy link
Member Author

@typescript-bot pack this

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 26, 2021

Heya @andrewbranch, I've started to run the tarball bundle task on this PR at a522672. You can monitor the build here.

@@ -1594,6 +1594,7 @@ namespace ts.Completions {

function tryGetImportCompletionSymbols(): GlobalsSearch {
if (!importCompletionNode) return GlobalsSearch.Continue;
isNewIdentifierLocation = true;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I discovered this was missing only because I added fourslash server tests... unrelated to main change

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 26, 2021

Hey @andrewbranch, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/103993/artifacts?artifactName=tgz&fileId=F2B1AE3F150B4136300EC26DC67EDD4AD9B375BB42E88F07E388F3F0E40B84BF02&fileName=/typescript-4.4.0-insiders.20210526.tgz"
    }
}

and then running npm install.


There is also a playground for this build and an npm module you can use via "typescript": "npm:@typescript-deploys/pr-build@4.4.0-pr-44259-6".;

@sheetalkamat
Copy link
Member

The host Project was creating an empty symlink cache too early—by deferring until after program creation, it can seed the cache with the symlinks discovered during module resolution for the files in that program.

I have concern with this.. useSourceOfProjectReferenceRedirect uses the symlink cache (?) to determine symlinked output to source file mapping (figuring out if file should exist and module should resolve).
How come that is not broken by this change.. are we missing some type reference directives to be added to the cache in that scenario?

@andrewbranch
Copy link
Member Author

My guess is it’s not broken because if we fail to cache or throw away that initially cached info, we eventually fall back to realpath if we need it. Maybe we need a way of merging the information from handleDirectoryCouldBeSymlink with what discoverProbableSymlinks provides. I’ll investigate more—I feel like the hardest part is coming up with failing tests here.

@sheetalkamat
Copy link
Member

With outDir and file not being built in referenced project might be good scenario to look at (with symlinks obviously) ?

@Jack-Works
Copy link
Contributor

this version fixes the problems in our project. thanks!

@andrewbranch andrewbranch requested a review from sheetalkamat May 27, 2021 23:01

if (!preferSymlinks) {
// Symlinks inside ignored paths are already filtered out of the symlink cache,
// so we only need to remove them from the realpath filenames.
const result = forEach(targets, p => !(shouldFilterIgnoredPaths && containsIgnoredPath(p)) && cb(p, referenceRedirect === p));
if (result) return result;
}
const links = host.getSymlinkCache
? host.getSymlinkCache()
: discoverProbableSymlinks(host.getSourceFiles(), getCanonicalFileName, cwd);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While refactoring for better preservation of a single SymlinkCache instance, I decided to get rid of this fallback. ModuleSpecifierResolutionHost is internal and all our implementations of it implement getSymlinkCache. It always delegates to Program.getSymlinkCache which delegates to Project.getSymlinkCache, but Program itself has a fallback. This should never be called directly by public API consumers, and even those who are using internal things should be taken care of via ts.createModuleSpecifierResolutionHost, so I think this is safe to remove.

Comment on lines +55 to +57
assert.deepEqual(
project.getSymlinkCache()?.getSymlinkedDirectories()?.get(link.path + "/" as Path),
{ real: "/packages/dep/", realPath: "/packages/dep/" as Path }
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This proves that directoryExists calls during program creation stayed in the cache because the /app/packages/tsconfig.json project has no source files with successful resolutions to the symlinked project, but it did discover the symlink as it tried to resolve import "dep/does/not/exist".

@maxdavidson
Copy link

maxdavidson commented Jun 3, 2021

We had an issue in TS 4.3 with auto imports suggesting long relative paths in a monorepo using Yarn workspaces. This PR seems to fix it! 🎉

TS 4.3.2
Skärmavbild 2021-06-03 kl  15 53 50

TS 4.2.4, this PR
Skärmavbild 2021-06-03 kl  16 34 10

@andrewbranch andrewbranch requested review from sandersn and amcasey June 7, 2021 18:35
@andrewbranch andrewbranch merged commit 703c1bc into microsoft:main Jun 8, 2021
@andrewbranch andrewbranch deleted the bug/pnpm-type-reference-directives branch June 8, 2021 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants