Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Add npm workspaces #103

Closed
wants to merge 11 commits into from
Closed

Conversation

ruyadorno
Copy link
Contributor

@ruyadorno ruyadorno commented Feb 14, 2020

npm workspaces

Summary

Add a set of features to the npm cli that provide support to managing multiple packages from within a singular top-level, root package.

Motivation

This feature has been requested by the community for a long time. The primary motivation behind this RFC is to fully realize a set of features/functionality to support managing multiple packages that may or may not be used together.

The name “workspaces” is already established in the community with both Yarn and Pnpm implementing similar features under that same name so we chose to reuse it for the sake of simplicity to the larger community involved.

Detailed Explanation

After sourcing feedback from the community, there are 2 major implementations/changes required in the npm cli in order to provide the feature set that would enable a better management of nested packages.

  • Make the npm cli workspace-aware.
  • Install: In a npm workspaces setup users expect to be able to install all nested packages and perform the associated lifecycle scripts from the Top-level workspace, it should also be aware of workspaces that have a dependency on one another and symlink them appropriately.

The set of features identified in this document are the ones that are essential to an initial MVP of the npm workspaces support. The community should expect further development of this feature based on the feedback we collected and documented at the end of this RFC.

Rationale and Alternatives

First and foremost there’s the alternative of leaving the problem set for userland to solve, there’s already the very popular project Lerna that provides some of these features.

Also available is the alternative of supporting only the install (or bootstrap as Lerna names it) aspect of this proposal, following a less feature-rich approach but one that would still enable the basic goal of improving the user experience of managing multiple child packages but from all the feedback collected during the research phase of this RFC, this alternative is much less desirable to the community of maintainers involved.

Implementation

1. Workspaces configuration: Making the npm cli workspace-aware

We're following the lead of Yarn in supporting the workspaces package.json property which defines a list of paths, each of these paths may point to the location of a workspace in the file system but it also support globs.

package.json example:

{
    "name": "workspace-example",
    "version": "1.0.0",
    "workspaces": {
        "packages": [
            "packages/*"
        ]
    }
}

package.json shorthand example:

{
    "name": "workspace-example",
    "version": "1.0.0",
    "workspaces": [
        "packages/*"
    ]
}

The npm cli will read from the paths and globs defined in this workspaces configuration and look for valid package.json files in order to create a list of packages that will be treated as workspaces.

Note: The packages property should support familiar patterns from npm-packlist files definition such as negative globs.

2. Installing dependencies across child packages

Change npm install (arborist) behavior to make it properly install dependencies for every workspace defined in the workspaces configuration described above.

Arborist should also be aware of all workspaces in order to correctly link to another internal workspace should it match the required semver version of an expected dependency anywhere in the installing tree. e.g:

// Given this package.json structure:
├── package.json { "workspaces": ["dep-a", "dep-b"] }
├── dep-a
│   └── package.json { "dependencies": { "dep-b": "^1.0.0" } }
└── dep-b
    └── package.json { "version": "1.3.1" }

$ npm install

// Results in this symlinking structure:
├── node_modules
│   ├── dep-a -> ./dep-a
│   └── dep-b -> ./dep-b
├── dep-a
└── dep-b

For the initial workspaces implementation, we're going to stick with arborist's default algorithm that privileges hoisting packages but will place packages at nested node_modules whenever necessary.

Examples

Expanding on symlinking structure and package-lock file shape.

Given a npm workspaces setup with the following contents:

$ cat ./package.json
{
    "name": "foo",
    "version": "1.0.0",
    "workspaces": [
        "./core/*",
        "./packages/*"
    ],
    dependencies: {
        "lodash": "^4.x.x",
        "libnpmutil": "^1.0.0"
    }
}

$ cat ./core/libnpmutil/package.json
{
    "name": "libnpmutil",
    "version": "1.0.0",
    "dependencies": {
        "lodash": "^4.x.x"
    }
}

$ cat ./packages/workspace-a/package.json
{
    "name": "workspace-a",
    "version": "1.7.3",
    "peerDependencies": {
        "react": "^16.x.x"
    },
    "dependencies": {
        "workspace-b": "^2.0.0"
    }
}

$ cat ./packages/worskpace-b/package.json
{
    "name": "workspace-b",
    "version": "2.1.1",
    "peerDependencies": {
        "react": "^16.x.x"
    }
}

$ cat ./packages/workspace-c/package.json
{
    "name": "workspace-c",
    "version": "1.0.0",
    "peerDependencies": {
        "react": "^16.x.x"
    },
    "dependencies": {
        "workspace-b": "^1.0.0"
    }
}

Will result in the following symlinking structure:

$ tree
.
├── package-lock.json
├── node_modules
│   ├── lodash
│   ├── libnpmutil -> ./core/libnpmutil
│   ├── workspace-a -> ./packages/workspace-a
│   ├── workspace-b -> ./packages/workspace-b
│   ├── workspace-c -> ./packages/workspace-c
│   └── react
├── core
│   └── libnpmutil
└── packages
    ├── workspace-a
    ├── workspace-b
    └─ worspace-c
        └── node_modules
            └── workspace-b@1.0.0

And the following package-lock.json files:

NOTE: The following lockfile is for illustration purpose only and its final shape might differ.

$ cat ./package-lock.json
{
  "name": "foo",
  "version": "1.0.0",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "name": "foo",
      "version": "1.0.0",
      "dependencies": {
        "lodash": "^4.17.15",
        "libnpmutil": "^1.0.0"
      }
    },
    "core/libnpmutil": {
      "name": "libnpmutil",
      "version": "1.0.0"
    },
    "node_modules/lodash": {
      "name": "lodash",
      "version": "4.17.15",
      "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.15.tgz",
      "integrity": "sha512-8xOcRHvCjnocdS5cpwXQXVzmmh5e5+saE2QGoeQmbKmRS6J3VQppPOIt0MnmE+4xlZoumy0GPG0D0MVIQbNA1A=="
    },
    "node_modules/libnpmutil": {
      "resolved": "core/libnpmutil",
      "link": true
    },
    "node_modules/workspace-a": {
      "resolved": "packages/workspace-a",
      "link": true
    },
    "node_modules/workspace-b": {
      "resolved": "packages/workspace-b",
      "link": true
    },
    "node_modules/workspace-c": {
      "resolved": "packages/workspace-c",
      "link": true
    },
    "packages/workspace-a": {
      "name": "workspace-a",
      "version": "1.7.3"
    },
    "packages/workspace-b": {
      "name": "workspace-b",
      "version": "1.0.0"
    },
    "packages/workspace-c": {
      "name": "workspace-c",
      "version": "1.0.0"
    }
  },
  "dependencies": {
    "lodash": {
      "version": "4.17.15",
      "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.15.tgz",
      "integrity": "sha512-8xOcRHvCjnocdS5cpwXQXVzmmh5e5+saE2QGoeQmbKmRS6J3VQppPOIt0MnmE+4xlZoumy0GPG0D0MVIQbNA1A=="
    },
    "libnpmutil": {
      "version": "file:core/libnpmutil"
    },
    "workspace-a": {
      "version": "file:packages/workspace-a"
    },
    "workspace-b": {
      "version": "file:packages/workspace-b"
    },
    "workspace-c": {
      "version": "file:packages/workspace-c",
      "dependencies": {
        "workspace-b": {
          "version": "1.0.0",
          "resolved": "https://registry.npmjs.org/workspace-b/-/workspace-b-1.0.0.tgz",
          "integrity": "sha512-8xOcRHvCjnocdS5cpwXQXVzmmh5e5+saE2QGoeQmbKmRS6J3VQppPOIt0MnmE+4xlZoumy0GPG0D0MVIQbNA1A=="
        },
      }
    }
  }
}

Dictionary

During the discussions around this RFC it was brought up to our attention that a lot of the vocabulary surrounding what the larger JavaScript community understands as "workspaces" can be confusing, for the sake of keeping the discussion as productive as possible we're taking the extra step of documenting what each of the terms used here means:

  • npm cli: The npm cli 😉
  • npm workspaces: The feature name, meaning the ability to the npm cli to support a better workflow for working with multiple packages.
  • workspaces: A set of workspace.
  • workspace: A nested package within the Top-level workspace file system that is explicitly defined as such via workspaces configuration.
  • Top-level workspace: The root level package that contains a workspaces configuration defining workspaces.
  • workspaces configuration: The blob of json configuration defined within package.json that declares where to find workspaces for this Top-level workspace package.
  • dependencies: A set of dependency.
  • dependency: A package that is depended upon by another given package.
  • dependent: A package which depends on another given package.
  • symlink: A symbolic link between files.
  • globs: String patterns that specifies sets of filenames with special characters.
  • Arborist: The npm@7 install library
  • hoisting packages: Bringing packages up a level in the context of an installation tree.
  • scripts: Arbitrary and lifecycle scripts defined in a package.json

Prior Art

@ruyadorno ruyadorno added Agenda will be discussed at the Open RFC call Enhancement new feature or improvement semver:minor new backwards-compatible feature Release 7.x labels Feb 14, 2020
@ruyadorno ruyadorno added this to the OSS - Sprint 4 milestone Feb 14, 2020
@ruyadorno ruyadorno requested a review from a team February 14, 2020 20:34
Copy link
Contributor

@ljharb ljharb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a few primary use cases i'd like to see explicitly called out as supported:

  1. multirepo
  • The root is private: true, and only lists deps/devDeps/peerDeps that apply to repo-level maintenance
  • each subpackage lists every dep/devDep/peerDep that it needs, just as if it lived in a repo by itself
  • subpackages depended on by other subpackages are able to be linked for dev/tests, but also able to be unlinked, so the explicit versions referenced are tested
  • dep/peerDeps are always published with version numbers, never as links/file refs
  • subtle peer dep conflicts are surfaced, ideally based on the published refs and based on linked refs
  1. something custom (eg, eslint-plugin-import)
  • the root is a package that's published
  • subpackages also exist that are published (and npmignored by the root)
  • the rest is the same as in a multirepo
  1. monorepo
  • basically nothing is published, everything is private: true
  • the rest is the same as in a monorepo, except that everything's always linked because there's no published versions

accepted/0000-workspaces.md Outdated Show resolved Hide resolved
accepted/0000-workspaces.md Outdated Show resolved Hide resolved
@isaacs
Copy link
Contributor

isaacs commented Feb 15, 2020

the root is a package that's published

I'm really perplexed about a workspace root that's also a package getting published. How does that work when installing a workspace root as a dependency? Do we just strip off the workspaces declaration when installing, or something? Do the workspace packages get listed as dependencies?

@ljharb
Copy link
Contributor

ljharb commented Feb 15, 2020

presumably you’d want to do that anyways if you didn’t require private: true for workspaces to work.

see https://github.com/benmosher/eslint-plugin-import for that particular example.

Copy link

@wesleytodd wesleytodd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a bunch more things I hope to write up this weekend, but here is my first feedback.

accepted/0000-workspaces.md Outdated Show resolved Hide resolved
accepted/0000-workspaces.md Outdated Show resolved Hide resolved

### 4. Publishing workspaces

A workspace may not be published to the registry and for all publishing purposes having a valid `"workspace"` entry in a `package.json` is going to be the equivalent of `"private": true`.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a needless restriction and blocks some interesting use cases. I can imagine a developer experience which involves teams publishing workspaces as a mechanism for sharing a setup. You could just specify that installing a published package with a workspaces key will do nothing, and then let userland tooling extend this if it wants.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@isaacs

Do we just strip off the workspaces declaration when installing

I would think for a use case like sharing a setup you would not strip it, only ignore it.

Do the workspace packages get listed as dependencies?

No, as with other workspace implementations you only get it when developing locally with it as the top level package. So to publish a package it must define all that it requires when installing from the registry.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if it's not a top of tree module (root or link target, not a deep in node_modules) then the workspaces object is irrelevant? That could work I guess.

Copy link
Contributor Author

@ruyadorno ruyadorno Feb 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wesleytodd this is looking interesting 👍 assuming there's no technical limitations that might prevent it - I'm onboard with removing the restriction

You could just specify that installing a published package with a workspaces key will do nothing, and then let userland tooling extend this if it wants.

just to clarify, in this case there's no filtering out mechanism at the moment of publishing a top-level workspace 🤔 meaning all the contents of nested packages within it get published along with that top-level package?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's the right call to make a "workspaces" field in package.json equivalent to "private": true. "Sharing configuration" (for an identical workspace, what?) should be a separate concern.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

meaning all the contents of nested packages within it get published along with that top-level package?

My Yeah you would use the normal methods for specifying which files make it into the tarball if you wanted different behavior.

Sharing configuration" (for an identical workspace, what?) should be a separate concern.

Just wondering, why do you think this? To me this clause of "workspaces means private" makes it a concern here. I am fine having that as a separate conversation as long as this clause does not make it into the implementation.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wesleytodd The manifest that coordinates the workspaces, as I've commented elsewhere, should not be conflated with a manifest that specifies a publishable package. The fact that Lerna can do that is an accident of history, and does not represent what I believe to be a robust pattern for managing multi-package repositories.

Copy link

@wesleytodd wesleytodd Feb 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Publishing it is a distribution mechanism, and the registry is the distribution mechanism in our ecosystem. If folks find novel ways to use this as both a distribution mechanism and a tool to build better developer experience, GREAT! I don't see this needless restriction gaining us anything at this time.

While it might not not help your monorepo-centric tool doesn't mean that it cannot help other types of setups. Do you think that your worries hold up when you move outside of a monorepo? Do you have suggestions for supporting multi project workspaces outside of a monorepo?

My single biggest concern for this implementation: that it not be tightly coupled to the monorepo setup. If npm just follows yarn and lerna on this I will continue not to be able to use them for my main use cases. Most OSS projects are a collection of projects in their own repos. Some popular ones outgrew the tooling (a separate problem I think we can solve), but the vast majority have not. If we do not make this a top priority for workspaces in npm I think we will have missed an opportunity to appeal to a much larger audience.

accepted/0000-workspaces.md Outdated Show resolved Hide resolved
@wesleytodd
Copy link

wesleytodd commented Feb 15, 2020

multirepo vs monorepo

I would really love for us to get past this focus on "monorepo" vs "multirepo". Can we just talk about Multi-Project support? The repo part is a vestige of our node module resolution mechanism being tied to the filesystem structure and a repo being the only reliable way of ensuring file structures across systems. In the future we will get a lot more value in decoupling this all from the filesystem layout, and building a workspace implementation by copying Yarn v1 is already behind the state of the art.

There are a bunch of examples of decoupling the file system from the module resolution mechanism: Yarn PnP, tink and Import Maps. I think we should strongly think about how multi-project looks when we have a future where the directory structure does not dictate where modules live on the filesystem.

@isaacs
Copy link
Contributor

isaacs commented Feb 15, 2020

There are a bunch of examples of decoupling the file system from the module resolution mechanism: Yarn PnP, tink and Import Maps.

Agreed! And they're all outside the scope of this rfc ;)

Our intent is to use a fallback fs implementation (ie, tink), so we still have to figure out what the file system will look like. If we then virtualize it, we still need a way to unwind it when requested and need to know what to virtualize. So, all this gross disk stuff needs to get worked out regardless. Also, files on disk has its advantages.

@isaacs
Copy link
Contributor

isaacs commented Feb 15, 2020

something custom (eg, eslint-plugin-import)
the root is a package that's published
subpackages also exist that are published (and npmignored by the root)

What happens if they're not npmignored? I think maybe we should treat them just like normal package contents, included by default.

@wesleytodd
Copy link

wesleytodd commented Feb 15, 2020

@isaacs your points are a bit unclear to me. I am not sure what a "fallback fs" means here, and since this whole proposal is centered around the existing disk layout based methodology then how is that not applicable?

As for "files on disk has its advantages", I don't disagree, but could you elaborate on how that applies specifically to this workspace implementation?

@isaacs
Copy link
Contributor

isaacs commented Feb 15, 2020

@isaacs your points are a bit unclear to me. I am not sure what a "fallback fs" means here, and since this whole proposal is centered around the existing disk layout based methodology then how is that not applicable?

Fallback fs is the approach used by tink. We model out the file system, and instead of providing an alternative module loader, like pnp does, we virtualization the fs module in node, such that any ENOENT in a node_modules folder falls back to the appropriate package contents in a single global cache.

On advantage of this approach is that you can always unwind to the "real" location, and no changes are required in the vast majority of use cases.

My point is that, even with such a virtualized system, we still need to decide the shape of it as a file system.

The biggest advantage of files on disk (or a sufficiently convincing fallback fs) is that it works trivially with the other tools, including editing to float patches, webpack and other transpilers, etc.

@wesleytodd
Copy link

Fallback fs is the approach used by tink

Ah, I hadn't looked at that part of what it was doing. I am not opposed to this approach but I would like to know why you think that is better than a loader as the runtime portion? It seems like this reaches out of the package installer's domain and is also more heavy handed.

On advantage of this approach is that you can always unwind to the "real" location, and no changes are required in the vast majority of use cases.

It seems like this is possible from any of the approaches we might take here, right?

we still need to decide the shape of it as a file system.

I guess since the "shape of it as a filesystem" maps directly to the shape of the optimized dep tree? Because if you choose something like an Import Map over a virtualized fs that relationship doesn't really matter anymore.

The biggest advantage of files on disk (or a sufficiently convincing fallback fs) is that it works trivially with the other tools, including editing to float patches, webpack and other transpilers, etc.

This is a great point and is the thing I have always been worries about with tink and PnP. Are those other approaches finding a solution to this? I do think we should have an answer for this as it relates to workspaces so the design of that doesn't have to change too much in relation.

ruyadorno and others added 2 commits February 17, 2020 12:16
Co-Authored-By: Wes Todd <wes@wesleytodd.com>
Co-Authored-By: Wes Todd <wes@wesleytodd.com>
@ruyadorno
Copy link
Contributor Author

@ljharb having the final implementation being flexible enough to support these use cases is my #1 priority 😁 that said, I have a few questions regarding some specific examples:

  • subpackages depended on by other subpackages are able to be linked for dev/tests, but also able to be unlinked, so the explicit versions referenced are tested

by "able to be linked for dev/tests" are you suggesting anything more complex than just the symlinking generated after npm install that is currently described currently in the RFC?

"also able to be unlinked, so the explicit versions referenced are tested" this sounds to me like something that might be outside of the scope of an initial workspaces implementation but to be clear, in the current proposal the way to unlink a subpackage (or have it be fetched from the registry instead) would be to define a semver range that is not satisfied by the package.json version that is currently sitting in the file system in its dependents, but again all this is handled by arborist meaning that if you want to do things for dev/tests only it would imply a more complex workflow in which you have to manually edit versions, run npm install and test things again - does this makes sense? did I understood correctly your points in this item? 😊 let me know

  • subtle peer dep conflicts are surfaced, ideally based on the published refs and based on linked refs

This item seems to be outside of the scope of this initial work 🤔 at least in its current form - how could we better handle that scenario? Can you (or anyone else really) provide some examples/ideas?

  • subpackages also exist that are published (and npmignored by the root)

Are these manually npmignored?


Feel free to suggest changes to the document if you want to see any specific wording in the final ratified version.

And thanks for the list! 😄 it provides a great initial checklist for all test cases we'll want to make sure to have in arborist.

@ljharb
Copy link
Contributor

ljharb commented Feb 17, 2020

@ruyadorno

are you suggesting anything more complex

the current main flaw with linking is that every peer dep has to be separately linked too; these would need to be solved.

did I understood correctly your points in this item?

What I mean is, there's two modes that are important: development (where links are ideal) and CI/production (where i want to test what consumers will get when they install). In other words, if subpackage A depends on subpackage B@2.0.0, i'd expect in dev that B would be the current working directory, but that when "unlinked", or in prod, that it'd be the published v2.0.0.

This item seems to be outside of the scope of this initial work 🤔 at least in its current form - how could we better handle that scenario? Can you (or anyone else really) provide some examples/ideas?

Whether it's in scope or not, I'd consider it a blocker. Issues around peer deps are wildly multiplied in a workspaces scenario - for example, enzyme has a handful of "react adapters", each of which requires a different, conflicting version of react. In airbnb's javascript style guide, there's two eslint config packages (the main one, and the base), the main requires the base, and both the main and the base peer depend on eslint as well as a number of eslint plugins, some overlapping. Basically, my concern is that a workspaces development model makes it much easier to obscure issues around dependency conflicts, and I think it's highly critical that the tooling surface these.

Are these manually npmignored?

Yes, I think it's totally fine to leave "what's in the tarball" management to the user, but it might be nice to warn the user if one published package would end up containing another.

@evocateur
Copy link

the root is a package that's published

FWIW, I really don't like this pattern. The root manifest of a multi-package repository should be the place you coordinate workspaces, test/dev dependencies, and other bits of "global" configuration. Making the root manifest also a published package is not gaining enough benefits to outweigh the drawbacks (confusing, arbitrary ignores) in my opinion.

@wesleytodd
Copy link

wesleytodd commented Feb 18, 2020

Just to hoist this topic to the top level thread:

I strongly think we need to think both about monoreps (well trodden ground today) as well as multirepo.

Most projects I work on are multirepo. We have great reasons for this and we will not be changing. Today our tooling provides almost no support for this, and so I have many DX workarounds. To be clear, I am not against monorepos. I helped move my last companies codebase into one, it is great for many types of projects. But not all. We can do better for this majority of package authors, and we should.

@ljharb
Copy link
Contributor

ljharb commented Feb 18, 2020

@evocateur re #103 (comment)

FWIW, I really don't like this pattern

ftr neither do i, but the best way for repos that use it to migrate to a better pattern is if they have reliable tooling available that supports both patterns - like npm.

Co-Authored-By: Wes Todd <wes@wesleytodd.com>
@darcyclarke darcyclarke added this to the OSS - Sprint 6 milestone Mar 10, 2020
@isaacs
Copy link
Contributor

isaacs commented Mar 18, 2020

Work is progressing on this over here: npm/arborist#50

@lmcarreiro
Copy link

From those three use cases that @ljharb pointed out at the beginning, the third one is the most important:

  1. monorepo
  • basically nothing is published, everything is private: true
  • the rest is the same as in a monorepo, except that everything's always linked because there's no published versions

The first two are already supported by third party tools, like Yarn, Lerna and Microsoft Rush.

Breaking a private big project into a bunch of small private pieces is way more common than breaking it into public packages. And yet we don't have a simple solution to such a simple problem. People don't do this in Javascript projects because there is no simple way to do that. So in any big project with a single package.json, every new installed dependency are available on the entire source code, which is a nightmare from the point of view of software architecture.

During the install, if some dependency is a workspace, it should be handled differently if it is a local dev install npm install or a production install npm install --production, and if it is private or not:

  • On a dev machine, without --production: just create symbolic links an it will work fine.
  • On a server, with --production: have to take a look at the local package.json file
    • private: false: just fetch the dependency from the registry
    • private: true: pack the dependency and use the resulting .tgz, and if this dependency depends on other private workspaces, this should be done recursivelly

Yarn v2 introduced the workspace: protocol. It creates a link on a dev install, but on prod install it always tries to fetch from the registry, even when the dependency has private: true.

@remorses
Copy link

Is someone working on this? A lot of people would migrate from yarn as soon as this feature is implemented

@ruyadorno ruyadorno removed the Agenda will be discussed at the Open RFC call label Apr 17, 2020
@ruyadorno
Copy link
Contributor Author

ruyadorno commented Apr 17, 2020

@lmcarreiro that sounds like an interesting idea! I'll bring that up during the next OpenRFC call but as work on this RFC has been finalized it might be better to follow up with it in nodejs/modules#117

@remorses that'd be me 😬 you can follow up progress in the link @isaacs shared above or just follow up v7 progress in general at npm/arborist 😊

@ruyadorno ruyadorno closed this in 9e38cbc Apr 17, 2020
@ruyadorno
Copy link
Contributor Author

I noted the PR doesn't contain all the historic info on the ratification process, so just for the sake of documenting these details I'm going to leave these references here:

@smably
Copy link

smably commented Apr 29, 2020

I'm a little late to the party here, but reading the RFC it seems that there will be a single package-lock.json in the root. Is there a migration path for projects currently using Lerna where there is one package-lock.json file per leaf package? I.e., will there be a way to maintain the existing locked versions when introducing npm workspaces to a project managed by Lerna?

@ruyadorno
Copy link
Contributor Author

@smably that's a very good question I would love to hear @evocateur thoughts about it, since I believe there might be tooling out there from the Lerna community around it from the time Yarn v1 first shipped workspaces.

That said the idea is that once npm@7 is shipped, Lerna+npm users will now have the option to migrate to a useWorkspaces bootstrap setup similar to the way Lerna+Yarn users were already familiar with - migrating a project to that kind of setup might be non-trivial though, the mismatching package-lock.json problem you just described is a side-effect of a much larger change which is that now you only have a single installation tree and that can lead to problems like mismatching peer dependencies (the one I saw most often having tried a dozen different projects).

@smably
Copy link

smably commented Apr 30, 2020

@ruyadorno Makes sense. I don't think there's a way to merge multiple Yarn lockfiles because Yarn's lockfile format maps each semver range directly to a resolved version, so it's not possible to maintain different resolutions for the same semver range in different workspaces. E.g., one of my (pre-workspaces) packages could have in its yarn.lock

lodash@^4.17.5:
  version "4.17.5"
  resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.17.5.tgz#99a92d65c0272debe8c96b6057bc8fbfa3bed511"
  integrity sha512-svL3uiZf1RwhH+cWrfZn3A4+U58wbP0tGVTLQPbjplZxZ8ROD9VLuNgsRniTlLe7OlSqR79RUehXgpBW/s0IQw==

and another could have

lodash@^4.17.5:
  version "4.17.15"
  resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.17.15.tgz#b447f6670a0455bbfeedd11392eff330ea097548"
  integrity sha512-8xOcRHvCjnocdS5cpwXQXVzmmh5e5+saE2QGoeQmbKmRS6J3VQppPOIt0MnmE+4xlZoumy0GPG0D0MVIQbNA1A==

and there'd be no way to represent both of those with a single yarn.lock in the root.

Whereas in this RFC, if I'm reading the package-lock example correctly, it appears that each workspace has its own nested dependencies tree, so it would be feasible to merge multiple lockfiles and maintain the original resolutions within each workspace.

@ruyadorno
Copy link
Contributor Author

Whereas in this RFC, if I'm reading the package-lock example correctly, it appears that each workspace has its own nested dependencies tree, so it would be feasible to merge multiple lockfiles and maintain the original resolutions within each workspace.

Right right, the npm lockfile format preserves information about nested dedupped modules which makes it theoretically possible to have them merged but you would still need some manual intervention in order to handle peerDependencies and any top-level conflicts - either way that would be def outside of the scope of the work we're doing in the npm cli now 😊 awesome tooling for someone in the community to pick up though 😉 😉

ruyadorno added a commit to ruyadorno/arborist that referenced this pull request Apr 30, 2020
Introduces support to workspaces; adding ability to build an ideal tree
that links defined workspaces, storing and reading lockfiles that
contains workspaces info and also reifying installation trees properly
symlinking nested workspaces into place.

Handling of the config definitions is done via @npmcli/map-workspaces
module added.

refs:

- https://github.com/npm/rfcs/blob/ea2d3024e6e149cd8c6366ed18373c9a566b1124/accepted/0026-workspaces.md
- https://www.npmjs.com/package/@npmcli/map-workspaces
- npm/rfcs#103
isaacs pushed a commit to npm/arborist that referenced this pull request May 1, 2020
Introduces support to workspaces; adding ability to build an ideal tree
that links defined workspaces, storing and reading lockfiles that
contains workspaces info and also reifying installation trees properly
symlinking nested workspaces into place.

Handling of the config definitions is done via @npmcli/map-workspaces
module added.

refs:

- https://github.com/npm/rfcs/blob/ea2d3024e6e149cd8c6366ed18373c9a566b1124/accepted/0026-workspaces.md
- https://www.npmjs.com/package/@npmcli/map-workspaces
- npm/rfcs#103

PR-URL: #50
Credit: @ruyadorno
Close: #50
Reviewed-by: @isaacs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement new feature or improvement semver:minor new backwards-compatible feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.