Package workspaces

[jneem]

Adds support for packaging an entire workspace, even when there are dependencies between the crates. The generated packages should be identical to the ones produced by packaging and publishing the crates one-by-one in dependency order, but the main benefit of this PR is that the packages can be created and verified locally before anything is published.

The main mechanism is the one in #13926, where we create a temporary local registry that "overlays" the true registry. We "publish" the crates in the local registry, which enables lockfile generation and verification of the dependent crates.

This adds --registry and --index flags to cargo package. They act
much like the same arguments to cargo publish, except that of course
we are not actually publishing to the specified registry. Instead, these
arguments affect lock-file generation for intra-workspace dependencies:
when simultaneously packaging a crate and one of its dependencies, the
lock-file will be generated under the assumption that the dependency
will be published to the specified registry.

You can also publish a subset of a workspace using -p arguments. In this case, there will be an error unless the chosen subset contains all of the dependencies of everything in the subset.

Fixes #10948. Based on #13926.

Compatibility issue

This PR introduces a case where cargo package will fail where it did not before: if you have a workspace containing two packages, main and dep@0.1.0, where main depends on dep@0.1.0 and dep@0.1.0 is already published in crates.io then attempting to package the whole workspace will fail. To be specific, it will package dep@0.1.0 successfully and then fail when trying to package main because it will see the two different packages for dep@0.1.0. Note that cargo publish will already fail in this scenario.

This shouldn't interfere with crates.io running cargo package for each package to diff the .crate files

• This might interfere if someone tried to verify their "published" MSRV by running cargo package.

The failure could be avoided by changing the local overlay source to not error out if there's a duplicate package; see here. However, failing early has the advantage of catching errors early.

[rustbot]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @weihanglo (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

• @rustbot author: the review is finished, PR author should check the comments and take action accordingly
• @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

[rustbot]

⚠️ Warning ⚠️

• These commits modify submodules.

[bors]

☔ The latest upstream changes (presumably #14004) made this pull request unmergeable. Please resolve the merge conflicts.

[jneem]

I think this is ready for a look.

There's a fair amount of churn in the tests: for workspace packaging we need to finish all the packaging before we start verifying, and this changes the order of console messages. I pushed as much of the test churn as I could into the first commit.

[jneem]

@epage I think the latest round of comments have all been addressed now.

[epage]

@bors r+

[bors]

📌 Commit a2f0a9e has been approved by epage

It is now in the queue for this repository.

[bors]

⌛ Testing commit a2f0a9e with merge b5d44db...

[bors]

☀️ Test successful - checks-actions
Approved by: epage
Pushing b5d44db to master...