Discovering .ts files under node_modules resulting in them being compiled

[billti]

I uncovered this general issue while testing loading JavaScript files from node_modules, but realized (and confirmed) this applies to TypeScript files also.

Basically, when we search the node_modules folder, we look for all supported extensions and add them to the compilation. Currently this means that if we find a TypeScript file in a Node module (e.g. someone published their package including the source), then we compile this also.

For example, if module sausages was published as follows (with the original source besides the compiler files):

 - node_modules
 |- sausages
   |- package.json
   |- core.ts
   |- core.js

Then we get the benefit of loading the .ts and getting the types from this. However on compiling an app which uses the sausages/core module, with outDir: "out", I get the resulting output (notice how core.js from the sausages package has been compiled into a node_modules folder in my outDir).

I already added a flag to denote if a file was found by searching Node modules in this pull request. I can refactor this to eliminate this issue if that seems reasonable.

[weswigham]

I think TS has had this issue since #4911 - there's some discussion in the PR on the subject. I also had changes in #4913 which marked files which were included via a node_modules directory (so they could be scoped appropriately).

[billti]

I've added a potential fix for this into #6928, as both depend on knowing if code was discovered downstream of a search under node_modules.

[igabesz]

I think this is a kinda serious issue in node based development. Currently I see the following options when using tsc in Node-based apps:

1. Include each *.ts files in your tsconfig. Better have a plugin for this (e.g. atom-typescript) because you don't wanna do this manually. The tsconfig will be updated whenever a TS file is added/deleted/replaced, which is also inconvenient.
2. Use exclude in tsconfig and exclude node_modules. If you have a TS-based module (e.g. you wrote one) then the included definitions won't be found (e.g. "typings": "lib/index.d.ts" in package.json) and neither can you use typings because it says you already have the typings under node_modules. Thank you.
3. Use exclude but NOT exclude node_modules: every *.ts file will be compiled under node_modules resulting in lots of warnings and errors because of using different linter options, not having all the typings used in those modules, etc.

Or leave tsc out of the business and let the IDE make the compilations (again, with atom-typescript, using filesGlob property which is the most convenient thing so far) and put all JS files into source control as well. It works conveniently on the developer side, but I don't think I have to enumerate the cons...

I know, there will be much rejoicing when global includes arrive #1927 -- but what until then??

> tsc -v
Version 1.8.9

[jpsfs]

I'm also facing this issue. My use case is the following:

| -- Solution
      | -- projectA
      | -- projectB
            | -- node_modules
                  | -- projectA (syslink to the folder above)

When compiling projectB, without the --outFile flag all works exactly as it should, unfortunately when using --outDir in conjunction with --module it bundles more than it should, bringing the code from projectA over to the bundle of projectB. Not what we were looking for.

Anyone knows a workaround for this?

[mhegazy]

in projectA's package.json, define `"typings": "<main .d.ts file of projectA>"``. this will force the compiler to load the .d.ts instead of loading the .ts files.

[jpsfs]

Thanks @mhegazy!

Although in this case we don't have a "main" .d.ts file.
In projectB we have several direct imports:

import "projectA/components/grid";
import "projectA/components/combobox";
...

And I rather not maintain a separate file that exports all of the components. We want to avoid giving the developers the possibility of doing something like

import * as PA from "projectA/components"

because each component may load additional 3rd party libraries and importing all will also load all of the dependencies, when most of the times they are not needed.

Do you have any other suggestion for this case?

Regards,

[mhegazy]

consider not including the .ts files in your package when you publish it.

[jpsfs]

@mhegazy we face this during development.

To give you a bit more context: the same team is responsible for the development and maintenance of more than a dozen typescript projects. To ease the development, the packages are linked, through npm link, in the development environment.

So, removing the .ts files when we publish them is fine (we are already doing that) but it's not an option during development.

The current setup works but the compiler performance is a serious handicap. On the top layer packages (those which depend on 8 or 9 other typescript packages) it seems the compiler goes through all the files and compiles them.

I already had the chance to talk with @jmatthiesen about our problems with VS and VSCode. I'm available to show you guys our environment if you are interested.

Best,

[mhegazy]

i see. thanks for the clarification. Do not think there are other workarounds other than fixing this issue.

[ps2goat]

+1 - I only see this when bundling my files using the outFile parameter. It doesn't pull in everything, though. Angular 2 (beta.08) is not being compiled and bundled with our app code, but some other third-party packages are. E.g., ng2-bootstrap (1.0.1-beta.2).

node_modules is already excluded in tsconfig.

[jpsfs]

@asfasdfasdfasdf angular2 is not bundled because the compiler is finding
the .d.ts files and not the actual .ts.

Currently we use TSC to bundle our code with the -outFile option but we
apply a custom post-processing step to remove the unwanted dependencies.
Em 19/05/2016 23:29, "asfasdfasdfasdf" notifications@github.com escreveu:

+1 - I only see this when bundling my files using the outFile parameter.
It doesn't pull in everything, though. Angular 2 (beta.08) is not being
compiled and bundled with our app code, but some other third-party packages
are. E.g., ng2-bootstrap (1.0.1-beta.2).

node_modules is already excluded in tsconfig.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#6964 (comment)

[mhegazy]

Fixed by #7075

[billti]

Actually, #7075 didn't resolve this. As I commented at #7075 (comment)

It tracks if a JavaScript file was loaded from searching node_modules so it knows not to compile it (a minor tweak to this logic could also solve #6964 if desired).

If you like, I can quickly add the change to resolve this issue later this evening. I just didn't want to conflate the two and risk not getting in the other fix if the were concerns on this issue (as there had been some discussion above). Let me know.

[ericbf]

@billti What about source files found under bower_components? Those seem to still be compiled and moved into the outDir. Should compilation exclude any files found under directories excluded with the "exclude" option of the tsconfig?

[jasonswearingen]

@mhegazy this bug still exists in the visual studio 2015 plugin (the 2.0 release). For example if you have the base64url npm package installed visual studio will (after a few minutes) show hundreds of duplicate identifier errors (false-positives) . It seems that the Visual Studio virtual projects crawl the node_modules. This is a problem because the base64url npm package bundles a typings folder with a .d.ts for node.

[mhegazy]

This is mainly about compiling them not about finding them.

[mhegazy]

Are these .ts files being compiled with your project?

[jasonswearingen]

they are not being compiled. I use a tsconfig.json, and if I run tsc it compiles fine. choosing "build" in visual studio also compiles fine.

but the auto-error-detecting service in visual studio gets confused by the node_modules/base64url/typings/globals/node/index.d.ts file and shows hundreds of errors from it mistakingly colliding with other node definitions.

[jasonswearingen]

this is also probably related to npm link modules having similar problems, but that also impacts tsc.

[AngeloSalvade]

That the tsc command generated the js files of ts sources in node_modules folders was for us actually a feature and not a bug! This worked fine in 1.8.10 but no longer in 2.0.3.

Our npm packages didn't contain the compiled js files, only the ts source files.
So it was very easy to compile the whole application with all its npm packages with one tsc call.
So the tsconfig.json file of the application was also applied to all the dependencies. This had the benefit that we could enforce that all the dependencies used the same compile options and the decision which options should be applied to the dependencies could be postponed until application build time.

Any chance that this "bug" could be reopenend and enabled again? Maybe optionally with a tsc option ?

[billti]

@AngeloSalvade you can still explicitly include those source files via the various include/exclude files settings in tsconfig.json if you want them part of the program. That should work fine.

[AngeloSalvade]

I already tried this and included "node_modules/*/". There are no errors. However, there are no generated JavaScript files for the TypeScript sources in the node_modules folder.

I used "module": "commonjs".
Is there some additional thing that I have to do?

[AngeloSalvade]

@billti I've provided a very short project https://github.com/softappeal/demo6964 showing my setup.
It would be very kind if you'd have a quick look at it and tell me what I'm doing wrong.

[billti]

I took a quick look. Modules resolved by searching under node_modules are excluded by default, so as mentioned, you need to include them explicitly. Take a look at the include and exclude settings of the test case at https://github.com/Microsoft/TypeScript/blob/master/tests/cases/projects/NodeModulesSearch/maxDepthExceeded/tsconfig.json . This compiles the source code it finds under node_modules - except for the module named m2. Something like this (without the allowJs setting) should work for you.

[AngeloSalvade]

I updated the project with your suggestion. It still doesn't work.
Could you please try it out? Does it work for you?

Setting maxNodeModuleJsDepth to a higher value or setting outDir also didn't help.

[billti]

Looks like you're right - I just tried it and can't get it to work. I'll need to debug through it and figure out what's going on here. I already fixed this exact scenario explicitly once (see discussion history at #9542 ), and as I pointed to above, there is a test case to cover it, so I'll need to dig in deeper.

Thanks for reporting! Sorry for the inconvenience.

[billti]

OK. Looks like this is still as intended, and my memory is rusty. The compiler still defaults to excluding node_modules, bower_components, etc... if no specific exclude setting is provided (see https://github.com/Microsoft/TypeScript/blob/master/src/compiler/commandLineParser.ts#L934 ). You'll see in the test case I pointed to, an exclude property is given.

In fact, if you specify an exclude, then we default to all files in all folders (if files or include aren't given), so simply replace your line "include": ["**/*"] with "exclude": [], and that should fix it by including everything.

Another (separate) point which may be just in your sample, but for your module you have no main property, just typings, which works for the TypeScript resolution, but when running under node it won't be able to locate the module entry file (as your one module in the package is not the default index.js).

Hope this clears things up! I'll try and get it documented a little clearer.

[billti]

Note: Per our docs at http://www.typescriptlang.org/docs/handbook/tsconfig-json.html:

Files included using "include" can be filtered using the "exclude" property. However, files included explicitly using the "files" property are always included regardless of "exclude". The "exclude" property defaults to excluding the node_modules, bower_components, and jspm_packages directories when not specified.

That last sentence seems to cover it.

[AngeloSalvade]

@billti Thanks a lot for your fast help. After replacing include with exclude, it works as I hoped. I updated the project.
For me it wasn't that obvious that I had to exclude nothing to include node_modules ;-)

And yes I now about the main property. I just left it out because it was not important for showing the effect.