Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove failed to install optional modules from node_modules #2375

Merged
merged 4 commits into from
Jan 4, 2017

Conversation

lgeiger
Copy link
Contributor

@lgeiger lgeiger commented Jan 3, 2017

Summary

Optional Dependencies are first linked (files are copied from cache to node_modules) and then their install scripts are executed.
If install script fails this is considered OK but node_modules then has a half built folder.

Yarn now removes failed to install optional modules from node_modules.

Addresses #2274

Test plan

I manually tested this PR and it works as expected.
Unfortunately the integration tests fail but I can't figure out why. Any help would be really appreciated. The integration tests have been adjusted accordingly.

@lgeiger lgeiger force-pushed the cleanup branch 2 times, most recently from abf18e2 to 44b4e47 Compare January 4, 2017 00:06
@lgeiger lgeiger changed the title [WIP] Remove failed to install optional modules from node_modules Remove failed to install optional modules from node_modules Jan 4, 2017
Copy link
Contributor Author

@lgeiger lgeiger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is now ready to review 🎉

This behavior is important if one tries to rebuild native dependencies for the usage in Electron with electron-builder since it'll abort if the failing optional module is still present. For more informations see: electron-userland/electron-builder#1075

assert.ok(await fs.exists(path.join(config.cwd, 'node_modules', 'sub-dep')));
return runInstallWithoutCheck({}, 'should-install-failing-optional-sub-deps', (config) => {
assert.equal(fsNode.existsSync(path.join(config.cwd, 'node_modules', 'optional-failing')), false);
assert.equal(fsNode.existsSync(path.join(config.cwd, 'node_modules', 'sub-dep')), true);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sub-dependencies only required by the failing module still aren't deleted.

Copy link
Member

@bestander bestander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few questions.
Thanks for submitting the fix!

return runInstall({}, 'should-install-failing-optional-deps', async (config) => {
assert.ok(await fs.exists(path.join(config.cwd, 'node_modules', 'optional-failing')));
return runInstallWithoutCheck({}, 'should-not-install-failing-optional-deps', (config) => {
assert.equal(fsNode.existsSync(path.join(config.cwd, 'node_modules', 'optional-failing')), false);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you switch to existsSync from await fs.exists?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did it primarily out of two reasons:

  1. If I recall correctly fs.exists throws if the path is empty
  2. fs.exists is deprecated while fs.existsSync isn't

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Yarn we have our own wrapper to fs methods and fs.exists returns a Promise that is async/awaitable, it is not deprecated afaik.
For the async io operations we prefer using async/await everywhere, it is better to follow a single guideline for the project

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wrapped method uses Node's fs.exists underneath which is deprecated: https://nodejs.org/api/fs.html#fs_fs_exists_path_callback

That said I'm happy to change it for consistency 👍

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I did not know that.
We'll probably do a one-off codemod to change to fs.access everywhere

@@ -28,6 +28,18 @@ export const runInstall = run.bind(
[],
);

export const runInstallWithoutCheck = run.bind(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would you need install without check?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the optional-failing module will be removed from node_modules, check will fail.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That might be a problem, check should not fail after a correct install

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see. So just removing the folder from node_modules isn't enough and will cause check to fail since the module isn't installed.

@bestander
Copy link
Member

bestander commented Jan 4, 2017 via email

Copy link
Member

@bestander bestander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, the check algorithm needs a change to accept missing optional deps

@bestander bestander merged commit 70a412f into yarnpkg:master Jan 4, 2017
@lgeiger lgeiger deleted the cleanup branch January 4, 2017 13:22
@rgbkrk
Copy link

rgbkrk commented Jan 4, 2017

What's reflected in the yarn.lock if the optional dependency is not kept?

@bestander
Copy link
Member

Lock file represents full dependency graph.
If you install with --production it will still contain dev dependencies.
Same thing for failed optional dependencies

@rgbkrk
Copy link

rgbkrk commented Jan 4, 2017

Ok, that makes sense since many times the optional dependencies are platform specific modules (darwin, windows, linux, etc.). Thanks @bestander and @lgeiger!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants