cargo doc doesn't document dependencies that are using the semver trick

[carols10cents]

Problem

If a crate foo depends on a crate (say, an older version of rand) that uses the semver trick where an older version of rand depends on a newer version of rand such that the crate is an ancestor of an older version of itself in the dependency tree, then running cargo doc for foo will not include any docs for any version of rand.

What happens

• rand doesn't appear in the "Crates" sidebar of foo's documentation,
• There were no "Documenting rand" or rustdoc invocation lines for
  either version in cargo's verbose output
• There was no "output filename collision" warning

What I expected to happen

I expected rand to be in the sidebar and link to documentation similar to
what appears on https://docs.rs/rand/0.3.23/rand/index.html

Steps

Check out this repo, or perform these steps:

$ cargo new foo
$ cd foo
$ echo 'rand = "0.3.14"' >> Cargo.toml

The dependency tree is now:

$ cargo tree
foo v0.1.0
└── rand v0.3.23
    ├── libc v0.2.112
    └── rand v0.4.6
        └── libc v0.2.112

Then run:

$ cargo doc -v --open

and look at the sidebar in the generated docs; no rand will be present. Look
at the verbose cargo output; no rustdoc invocation for rand will be
present.

Possible Solution(s)

I believe this was introduced in:

• Fix some issues with cargo doc and the new feature resolver. #9077
• Fix panic with doc collision orphan. #9142

and it's a combination of removing older/smaller versions in favor of newer/larger versions (so rand 0.3.22 gets removed in favor of rand 0.4.0), which was added in the first PR, and then pruning orphaned units (rand 0.4.0 is orphaned because it was only a child of rand 0.3.22, which was removed), which was added in the second PR.

I tried working on a solution but I'm not sure the best way to fix it. I'm going to open a PR for the failing test I have at least.

Notes

Variations that don't reproduce the problem

• I am able to check out the rand repo commit 577d2a0 (there's no tag for
  0.3.23) and run cargo doc successfully. I do see the "output filename
  collision" warning and both versions being documented in cargo's verbose
  output. I do see rand 0.3.23 documented as expected because 0.3.23 appears
  to be consistently last and win the collision.
• If I check out commit 577d2a0 of rand locally, then in the foo crate add
  a path dependency instead of a registry dependency, I see the "output
  filename collision" warning. I do see rand in the sidebar and I see
  both versions being documented in cargo's verbose output. I do see rand
  0.3.23 documented as expected because 0.3.23 appears to be consistently last
  and win the collision.
• If I get 2 different major versions of a crate in my dependency tree but
  they're not in a hierarchical relationship, I don't get the "output filename
  collision" warning. The greater version number gets documented and appears in
  the sidebar.

Why this is important

I found out about this through an issue filed with the book.
The print versions of The Rust Programming Language instruct the reader to add
rand = "0.3.14" to their Cargo.toml, then run cargo doc --open and observe
that they now have the documentation for rand locally. This no longer works,
and I am unfortunately unable to change the paper already in circulation. 😢

I haven't run the stats on crates.io's database yet, but the semver trick is
widely known, recommended, and used, especially in popular crates.

I can also see people wanting the documentation of different versions
available in their local documentation; see this SO question
for what I think is an example. While using docs.rs and picking the version
you want to look at explicitly is probably a much more common workflow, it seems
more "correct" to offer local docs for each version of a crate in the dependency tree, which I know is complicated and related to rust-lang/rust#56169 and rust-lang/rust#61378.

Version

$ cargo version --verbose
cargo 1.57.0 (b2e52d7ca 2021-10-21)
release: 1.57.0
commit-hash: b2e52d7cab0a286ee9fcc0c17510b1e72fcb53eb
commit-date: 2021-10-21
host: aarch64-apple-darwin
libgit2: 1.3.0 (sys:0.13.23 vendored)
libcurl: 7.77.0 (sys:0.4.49+curl-7.79.1 system ssl:(SecureTransport) LibreSSL/2.8.3)
os: Mac OS 12.1.0 [64-bit]

[ehuss]

Thanks for the detailed writeup! Your analysis is spot on. I didn't consider this situation with the orphan check.

I've been trying to think of an easy way to address this, but everything I think of is either risky or complicated.

I'm wondering if it would be better to just try to do the proper fix. rustdoc needs a way for cargo to tell it where each crate is located. Then cargo could come up with some directory naming scheme to allow duplicates. I don't think it should be too difficult. However, some thought needs to be put in if --extern-html-root-url might be the right approach, or if something different is needed. There's also some uncertainty about how --extern-html-root-url should behave exactly (rust-lang/rust#82776 (comment) / rust-lang/rust#88646). And it would need to support relative paths I think.

[carols10cents]

I'm wondering if it would be better to just try to do the proper fix.

Yeah, I was afraid of that, but I think you're right, and doing that provides other benefits as well :)

[xapimaze]

As a newbie going through the book, I found the "work-around" from #10723 useful:
cargo doc -p $(cargo pkgid rand:0.3.23)

[dvankampen]

@carols10cents @xapimaze thank you for the details. Can someone provide a summary for those like myself introducing themselves to rust through the book, who experienced the error, and are curious what the issue is here? Is it related to the fact that rand was a binary "crate" (not sure on the term) vs. source-code based? Or something related to the way the version was described (which this bug title leads me to believe)?

And the -p argument suggested above - does that do something ONLY for the doc command, or does that do something to the crate version that would be used for the application? Because the workaround being "get documentation for a different version of the package than the one you're using" seems like it might lead to issues down the road.

Much appreciated!

[weihanglo]

Triage: I think we're waiting for a way to communicate with rustdoc. See #10241 (comment).

@rustbot label +S-blocked-external