yarn install overwrites linked modules with cached version #1214

lukebennett · 2016-10-18T22:43:56Z

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

Running yarn install for an app that has linked modules results in yarn copying its cached version of each module to the symlinked directory, overwriting the local copy in the process.

If the current behavior is a bug, please provide the steps to reproduce.

I struggled to reproduce this from scratch, it's possible that it's down to a specific combination of factors with my setup. This involves local copies of private/scoped npm modules yarn linked from an application, and referenced in package.json via file:../xxx. The version of the modules yarn has in its cache appears to be from when I first installed and ran it a few days ago (there have been numerous changes to the modules since), so it's possible the problem lies with its caching strategy as much as its installation strategy.

One observation I've made is that the problem occurs even when using yarn install --force to supposedly bypass the cache.

Happy to answer any questions and/or try anything to help pinpoint the problem.

What is the expected behavior?

yarn install should skip linked modules like npm intstall does. Performing the same steps above using npm install does not result in this problem.

Please mention your node.js, yarn and operating system version.

node v6.2.2
yarn v0.16.0
Mac OS X v10.10.5

The text was updated successfully, but these errors were encountered:

bmayen · 2016-10-20T00:06:33Z

This isn't specific to cached packages. I have a published package foo@1.0.0. Locally I link an unpublished package foo@2.0.0. My project has a package.json dependency for foo@2.0.0 and I link to that. When I run yarn after linking, it tries to install foo@2.0.0 from the repo instead of using the linked version and it fails because foo@2.0.0 hasn't been published yet.

jimdebeer · 2016-10-20T16:13:01Z

really need this

mstdokumaci · 2016-10-24T15:37:41Z

this is breaking

haggholm · 2016-10-27T17:59:38Z

This applies even when you install different packages. E.g.

cd @my/local-dep
yarn link
cd ../../@my/main-project
yarn link @my/local-dep
yarn add lodash

Even though I didn’t even explicitly do anything with @my/local-dep, yarn still goes through the modules and installs anything not locally installed, even if it’s linked.

This happens even if I specify --no-lockfile.

It’s a particularly nasty problem because it overwrites my local working copy of @my/local-dep, so it will overwrite anything you haven’t committed yet, effectively erasing local work.

$ yarn --version
0.16.1

joefiorini · 2016-10-28T14:07:14Z

I've noticed this issue as well. In my testing it seems to be limited to scoped modules, but that may not be true for everyone. My use case:

Our team works in a monorepo. To get shared packages locally, the only way to have an efficient development process is to symlink them. Since we can't save symlinked installs to package.json, we use file: specifiers with relative paths, followed by a library called linklocal that goes finds and symlinks local dependencies.

It's kind of a hack, but it works very well. However, npm shrinkwrap is so painful for us that switching to yarn is extremely advantageous, even if we have to work around a few warts due to early adoption.

This issue is very bad for that above workflow though. Anytime yarn tries to install we run the risk of losing work.

What exactly is the advantage of using the cache for local dependencies? Since they would never hit the network anyway, couldn't yarn just treat anything that's a file: reference as a cache miss and either skip copying it entirely (as npm does) or copy it in from the local dir?

haggholm · 2016-10-28T17:33:46Z

I’d be much happier if Yarn just left linked dependencies alone—if it’s a symlink, either do nothing on the assumption that I want to work with it as a symlink, or if for some reason that can’t be done, issue an error and refuse to do anything. Even deleting the symlinks and replacing them with normal dependencies would be better—still a nuisance, but at least not as bad as the current behaviour.

If I run yarn link, I should never have to worry about Yarn changing my files.
There’s no reason why Yarn should be modifying files outside of node_modules (and its own caches).

IMHO, the preferred behaviour would be to leave symlinked directories alone, unless there’s a version mismatch, in which case error out with a descriptive message.

bmayen · 2016-10-28T22:11:08Z

unless there’s a version mismatch, in which case error out with a descriptive message.

What would be the advantage of this? Why not allow linked packages to be any version? I believe that's how NPM currently works.

damonmaria · 2016-11-02T12:46:09Z

Having the same issue. It seems like it's replacing the linked module with the published version. As tho it's installing the package into the directory without realising it's a symlinked package. I don't think yarn's cache is the reason.

joefiorini · 2016-11-02T13:53:26Z

I was able to get around this issue for time being by using yarn link and then not adding the dependencies to my package.json at all. The script that starts our dev server & installs deps for CI/production builds also handles running yarn link for our shared packages, which ensures it happens everywhere we need to build our code.

wclr · 2016-11-12T00:41:33Z

This is very confusing behaviour of yarn, a big source of hassle, have you managed fix the logic behind it? I believe yarn should not check for cache when dealing with file: modules (it also refers to other non-versioned sources like remote git repo).

wclr · 2016-11-12T16:59:11Z

I've described inconsistent behaviour or yarn that leads to the issue: #1794

wclr · 2016-11-20T07:27:46Z

Btw in new 0.17.6 it seems that yarn doesn't replace content of simlinked folder (in node_modules) with version from cache, but installs new dependencies (that changed in linked package.json - which is nice)

Can others confirm that this not an issue anymore?
@lukebennett @joefiorini @haggholm

travisbloom · 2016-11-20T16:55:32Z

@whitecolor can confirm that with 0.17.6 the problem still persists. simlinked folder is having code overwritten by yarn.

travisbloom · 2016-11-20T16:58:27Z

This is happening with a privately scoped npm repo (@rocketrip/name-of-package)

wclr · 2016-11-20T20:06:57Z

@travisbloom right it seem to be true when you link scoped package, it works ok with not scoped symlinks (doesn't touch it).

@wyze can you look in to it, to close issues with scoped packages?

Scoped dependencies are nested one folder deeper than traditional dependencies, the possiblyExtraneous needed updating to work with the actual dependencies, not the entire scoped folder.

tgriesser · 2016-11-21T14:26:26Z

Just added a fix for this in #1970

DenysVuika · 2016-11-29T09:53:16Z

Still an issue with 0.17.9 and a huge blocker for most of the scenarios involving npm/yarn link.

rbilgil · 2016-11-30T14:13:21Z

+1, This is a big issue that caused us to lose uncommitted work from a linked repo. Would be good to get it fixed.

Scoped dependencies are nested one folder deeper than traditional dependencies, the possiblyExtraneous needed updating to work with the actual dependencies, not the entire scoped folder.

…1970) * Fix for #1214 Scoped dependencies are nested one folder deeper than traditional dependencies, the possiblyExtraneous needed updating to work with the actual dependencies, not the entire scoped folder. * Fix typo/variable renaming

AlastairTaft · 2017-02-08T03:36:49Z

Can confirm I don't get this issue in the latest version 0.19.1.

tgriesser · 2017-02-08T15:27:13Z

Yep, this can probably be closed now

AlastairTaft · 2017-04-11T00:02:48Z

I am still seeing this issue on yarn@0.22.0. I'm using 'lerna' so if I have packageA depending on packageB via a symlink and then run yarn upgrade packageB inside the packageA directory It ends up overwriting my non-commited local changes of packageB.

wclr · 2017-04-11T01:20:02Z

@AlastairTaft can you make a list of simple steps to reproduce?

dbpieter · 2017-04-27T13:32:48Z

@whitecolor

Simple reproduction with file: protocol.

Install a package through the registry (for example "lodash")
Change your package.json so that lodash uses file protocol (eg: "lodash": file:./custom_packages/lodash")
yarn install
observe node_modules => lodash version from cache is used instead of the one from custom_packages

yarn v0.22.0

wclr · 2017-04-27T14:09:02Z

@dalmaer and on the latest stable 23.2?

bestander · 2017-04-27T14:10:10Z

We just landed a fix to file: dependencies on master. Check out in 0.24 next week

…

On 27 April 2017 at 15:09, Alex ***@***.***> wrote: @dalmaer <https://github.com/dalmaer> and on the latest stable 23.2? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#1214 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACBdWJx3UiG6uRUHgZeNqqGlsUyGyd2Pks5r0KGIgaJpZM4KaX2W> .

oliversalzburg · 2017-05-04T13:10:57Z

This is still happening. I just opened #3288 to report the same behavior. Happens with 0.23.4. We're not using file protocol. Could possibly be related to scoped/private as that is where we're seeing it.

oliversalzburg · 2017-05-05T09:47:22Z

Is there any workaround for this which I am not aware of? Needless to say, having your working copies constantly overwritten by cached content without you realizing it has a major impact on development.

wclr · 2017-05-06T09:17:58Z

@oliversalzburg I advice to try this for development with local packages: https://github.com/whitecolor/yalc (it can make your life really easier)

oliversalzburg · 2017-05-06T18:11:56Z

@whitecolor Awesome. Thanks a lot. I'll check it out.

jonschlinkert · 2017-05-09T02:47:34Z

I'm still trying to confirm if this is what happened, but it appears that yarn overwrote local files across multiple files with cached symlinked versions. I'm using yarn 0.23.4

oliversalzburg · 2017-05-17T13:02:43Z

@bestander Can you reopen this issue? It is not solved. I would really appreciate some feedback on this.

bestander · 2017-05-17T13:04:38Z

@oliversalzburg please open a new one with a scenario how to reproduce it.
Ideally if you could provide a bash script to make it automated.

oliversalzburg · 2017-05-17T14:26:20Z

@bestander I had already opened #3288, but I'm having a really hard time reproducing the issue in isolation. It happens multiple times every day in our main project tree though. It would be really nice to investigate this with someone who is more familiar with the code base and maybe give me some pointers on what to look for.

danilo-drs · 2017-05-24T19:51:45Z

I'm getting this issue on version 0.24.5.
I have some packages that i developed and they are not published on any server, they are just linked.
My projects uses this linked packages and some published packages like express or bluebird.
when I try to run "yarn" to install all or a specific package it returns the error:

error Couldn't find package "package-name" on the "npm" registry

steps to reproduce:

cd path/to/a-dev-package
yarn link
cd path/to/a/project
yarn link a-dev-package
yarn add bluebird
it returns :
error Couldn't find package "a-dev-package" on the "npm" registry

bestander · 2017-06-15T22:31:36Z

Thanks, @danilo-drs, I've extracted this into a separate issue #3645

stelf · 2017-07-10T08:42:53Z

I believe this issue is not solved in v0.27.5. node here is 6.11 and npm 5.04.

Steps (note: A is a dependency to B in package.json):

yarn link / npm link module A (a symlink goes into the local $PREFIX/lib/node_modules/)
chdir into module B's dir
yarn link A (a symlink to A goes into B/node_modules/).
yarn install => complains about versions from npmjs.com, rather than taking into consideration the local

wclr · 2017-07-10T08:58:56Z

@stelf yarn takes into account what is in your package.json to keep consistency. So it actually doesn't take into account that you linked the module (there is no any notice about it, except symlink directory in node_modules ). There is link: dependencies in yarn, but it is more for linking local deps in monorepo. For working with external modules I successfully use yalc

fcoury · 2017-10-18T20:53:08Z

This is still happening with yarn v1.2.1:

> $ yarn link mongodb-promise                                                                                                                                         yarn link v1.2.1
success Using linked module for "mongodb-promise".
✨  Done in 0.07s.

> $ ls -lad node_modules/mongodb-promise                                                                                                                              
lrwxr-xr-x  1 fcoury  staff  21 Oct 18 18:52 node_modules/mongodb-promise -> ../../mongodb-promise

fcoury@iMac ~/code/faxing                                                                                                                                              > $ yarn                                                                                                                                                              [±master ●]
yarn install v1.2.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
✨  Done in 1.99s.

> $ ls -lad node_modules/mongodb-promise                                                                                                                              drwxr-xr-x  9 fcoury  staff  306 Oct 18 18:52 node_modules/mongodb-promise

So this is the expected behavior?

chaddjohnson · 2017-10-23T21:37:04Z

Still outstanding and needs to be fixed.

When I run yarn add lodash (for example) it removes any links I had set up for my project, and I have to manually run yarn link my-package-name again.

bestander · 2017-10-24T04:18:12Z

Looks like the original issue was fixed but there may be other cases when it is still relevant.
For everyone seeing this please open a new issue with the repro script.
It would be hard to reopen this one as it tracks too many things.

chaddjohnson · 2017-10-24T15:22:30Z

Created new issue #4770 as requested.

Daniel15 added the needs triage label Nov 4, 2016

wclr mentioned this issue Nov 7, 2016

Yarn overrides linked dependencies #1718

Closed

wclr mentioned this issue Nov 12, 2016

Inconsistent work with local (file:)/ remote (git) dependencies #1794

Open

tgriesser added a commit to tgriesser/yarn that referenced this issue Nov 21, 2016

Fix for yarnpkg#1214

4c95db9

tgriesser mentioned this issue Nov 21, 2016

Fix for #1214, linked scoped dependencies should not be overwritten #1970

Merged

This was referenced Dec 8, 2016

yarn-e2e/os=ubuntu-12.04 #29 failed #2188

Closed

yarn-e2e/os=ubuntu-16.04 #29 failed #2189

Closed

yarn-e2e/os=ubuntu-14.04 #29 failed #2190

Closed

mariusmarais mentioned this issue Dec 13, 2016

Yarn deletes contents of symlinked directories on install #2238

Closed

bestander closed this as completed Feb 8, 2017

jonschlinkert mentioned this issue May 17, 2017

yarn overwrites symlinked dependencies #3288

Closed

bestander mentioned this issue Jun 15, 2017

yarn link: Couldn't find package "package-name" on the "npm" registry #3645

Closed

bestander mentioned this issue Jul 5, 2017

yarn install fails for unpublished and linked-in dependency #2611

Closed

yarn install overwrites linked modules with cached version #1214

yarn install overwrites linked modules with cached version #1214

Comments

lukebennett commented Oct 18, 2016 • edited Loading

bmayen commented Oct 20, 2016

jimdebeer commented Oct 20, 2016 • edited Loading

mstdokumaci commented Oct 24, 2016

haggholm commented Oct 27, 2016

joefiorini commented Oct 28, 2016

haggholm commented Oct 28, 2016

bmayen commented Oct 28, 2016

damonmaria commented Nov 2, 2016

joefiorini commented Nov 2, 2016

wclr commented Nov 12, 2016 • edited Loading

wclr commented Nov 12, 2016

wclr commented Nov 20, 2016

travisbloom commented Nov 20, 2016

travisbloom commented Nov 20, 2016

wclr commented Nov 20, 2016 • edited Loading

tgriesser commented Nov 21, 2016

DenysVuika commented Nov 29, 2016

rbilgil commented Nov 30, 2016

AlastairTaft commented Feb 8, 2017

tgriesser commented Feb 8, 2017

AlastairTaft commented Apr 11, 2017

wclr commented Apr 11, 2017

dbpieter commented Apr 27, 2017

wclr commented Apr 27, 2017

bestander commented Apr 27, 2017 via email

oliversalzburg commented May 4, 2017 • edited Loading

oliversalzburg commented May 5, 2017

wclr commented May 6, 2017 • edited Loading

oliversalzburg commented May 6, 2017

jonschlinkert commented May 9, 2017

oliversalzburg commented May 17, 2017

bestander commented May 17, 2017

oliversalzburg commented May 17, 2017

danilo-drs commented May 24, 2017

bestander commented Jun 15, 2017

stelf commented Jul 10, 2017 • edited Loading

wclr commented Jul 10, 2017 • edited Loading

fcoury commented Oct 18, 2017 • edited Loading

chaddjohnson commented Oct 23, 2017

bestander commented Oct 24, 2017

chaddjohnson commented Oct 24, 2017

lukebennett commented Oct 18, 2016 •

edited

Loading

jimdebeer commented Oct 20, 2016 •

edited

Loading

wclr commented Nov 12, 2016 •

edited

Loading

wclr commented Nov 20, 2016 •

edited

Loading

oliversalzburg commented May 4, 2017 •

edited

Loading

wclr commented May 6, 2017 •

edited

Loading

stelf commented Jul 10, 2017 •

edited

Loading

wclr commented Jul 10, 2017 •

edited

Loading

fcoury commented Oct 18, 2017 •

edited

Loading