Stabilize -Z avoid-dev-deps as --minimal-cargo-lock

[infinity0]

Original issue: #4988
Implementation: #5012
Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#avoid-dev-deps
Issues: Z-avoid-dev-deps Nightly: avoid-dev-deps

-Z avoid-dev-deps flag avoids resolving dev-dependencies in certain targets like install.

@alexcrichton thinks that "in the long run [this wants] to be something like --minimal-cargo-lock which prunes all non-relevant dependencies like platform-specific dependencies that don't apply, dev-deps if you're not building tests, etc."

[matklad]

@infinity0

Clarification question: am I correct that cargo install currently avoids dev dependencies even if the flag is not specified? That is, this issue is specifically for cargo build command?

[infinity0]

@matklad when I last touched that code, yes. the dev_dependencies_no_check test is for this.

[ignatenkobrain]

@infinity0 do you have some patch for this?

We wanted to not package some optional dependencies and hit this issue.

[ignatenkobrain]

\cc @keszybz

[ignatenkobrain]

and also deps for disabled features, I hope. Right?

[infinity0]

@ignatenkobrain Cargo already carries this functionality as cargo build -Z avoid-dev-deps or cargo install (with no extra flags) but only for dev-dependencies, not optional dependencies or unused features or unused target-specific dependencies. No I don't have a patch for the other things I mentioned.

[ignatenkobrain]

@alexcrichton any chance to get someone to work on this? I'm pretty sure that all distributions which decide to package crates will get hit by this.

[phi-gamma]

What needs to be done in order to get his option stabilized?

Cargo pulling in Windows and Redox deps unconditionally is currently
forcing us to patch the Cargo.toml files of numerous crates. It would
remove a lot of friction if this behavior could instead be disabled globally.

[ehuss]

Sorry, at this time we don't have this particular feature on a path towards stabilization. We recognize that there are circumstances where it solves a problem, but we feel like the design hasn't been completely fleshed out, and nobody on the team has the time right now to shepherd that design.

[dkg]

fwiw, this looks like it needs to be resolved to fix problems with debian's autopkgtest suite and multiple crates.

[est31]

I think this is subsumed by #7916

[cuviper]

I think this is subsumed by #7916

It doesn't seem so. I tried by running cargo vendor >.cargo/config, then removing some of the dev-deps. I could still build with -Z avoid-dev-deps, but not with -Z features=dev_dep.

cargo 1.43.0-nightly (bda50510d 2020-03-02)

[est31]

Hmm right good point. I think the difference between the two features is that -Z features=dev_dep still compiles the dev dependencies, just doesn't unify features any more while -Z avoid-dev-deps does not compile them at all

[olekspickle]

Any chance this can take shape any time soon?
It's would greatly help with consolidating async\non-async stuff in domain-oriented crates IMO.

Right now I still have to build all async stuff just for tokio::test 😞

[SpriteOvO]

For people who are using stable version numbers to cargo check MSRV (so can't use nightly), there is a dirty hacky workaround.

[SteveLauC]

For people who are using stable version numbers to cargo check MSRV (so can't use nightly), there is a dirty hacky workaround.

Thank you for this workaround!

[epage]

Copying over from #14794 a description from an active user of this feature

#14794 (comment)

I tried to write something, I'm not really sure if I hit the correct abstraction level here at all, so please ask more questions if needed:

How does building binaries from rust source work in Debian:

Debian does not allow the build servers to fetch source code from the internet while building, instead all source code that generates a binary must be already packaged and part of Debian before the build of the binary can happen.

There is also a concept of source packages and binary packages in Debian, with rust this gets a tad complicated. For rust-libraries the debian-binary package contains the source code of the library (for C it would contain the .so file), but for rust-binaries the debian-binary package contains the binary.

Here is an example of the content of a Debian-binary package built from rust: https://packages.debian.org/sid/amd64/lsd/filelist

And here is the Debian-source package for that: https://packages.debian.org/source/sid/rust-lsd

For libraries this looks like this: https://packages.debian.org/sid/amd64/librust-futures-channel-dev/filelist

https://packages.debian.org/source/sid/rust-futures-channel

This means that Debian mirrors the dependency resolution system in cargo in it's own package system. The build then first does the dependency resolution on the Debian side, and installs librust--dev packages, these packages only contain rust source code that gets placed into /usr/share/cargo/registry/ .

Of note is also that the dependency resolution on Debians side is recursive, we read the content of Cargo.toml and only insert those entries into the dependency list, and then the apt dependency resolution will walk the tree and pull in what is needed.

Debian do not respect/care about the Cargo.lock file.

Testing of the source code is also slightly involved, tests in Debian are run in a separate section from the normal build, called autopkgtest, and we run those per feature plus all-features and no-features.

The dev-dependencies gets added as to the Depends: row for the autopkgtest section, but not the build section. One important aspect of why this is loops in the dependency tree, it's quite common that a -derive package dev-depends on the main package in rust and if those dev-dependencies got added to the build Depends in Debian we would end up with a loop in the build system that it can't handle.

#14794 (comment)

adding some more bits:

there's actually two different ways we test crate sources in Debian packaging:

* as part of the build using `cargo test`, if there are no dev-dependencies

* as part of autopkgtests, also using `cargo test`, if there are dev-dependencies

in both cases, the dev-dependencies are not used as part of the regular build, but just for the (decoupled, running after packages are built) autopkgtests.

for regular package building, we use a cargo wrapper that (among other things) injects -Z avoid-dev-deps for "build", "rustc", "doc", "test", "bench", "install" invocations, which means that the lack of dev-dependencies at package building time is not an issue. it can be an issue for other commands, especially those related to generating the source packages that get later build. for example, debcargo, our tool that translates Cargo.toml into debian packaging metadata, has support for packaging from a local crate source tree (in addition to the default mode, which uses a downloaded .crate archive from crates.io). in order to support such local crate sources, debcargo basically does the equivalent of cargo package, which in turn requires all dependencies to exist in the registry. if "the registry" is the local dir of packaged crates, then this requires all dependencies (including dev dependencies) to be installed, even though they are not used in any fashion at that point. similarly, other tools using/extending cargo might be unhappy if there is a cargo config pointing at the packaged crate "registry", as opposed to crates.io, since the former only offers an "incomplete" view.

so in summary, it would be great if cargo could be taught to ignore referenced crates not existing for various actions - it would make our packaging life easier. it's not required to build and test packages though, as that part is already handled by the existing options in cargo.

(similarly, it would be great to be able to tell cargo to ignore the non-existence of dependencies for targets that are not currently being looked at/built for/.., as we have to patch such things out, and it causes a lot of meaningless patch churn when pulling in updates - but that is probably a separate issue ;))

[Fabian-Gruenbichler]

like the copied comments above indicate, Debian heavily relies on this flag to build packaged crates (and only uses the dev-dependencies in the CI tests running using built packages).