rustdoc URL conflict resolution

[GuillaumeGomez]

cc @rust-lang/rustdoc

Rendered

[shepmaster]

Won't this break almost every link, like https://doc.rust-lang.org/std/string/struct.String.html? Seems like that should be listed as a drawback.

[Manishearth]

Yes, this is my primary concern here. We'd be breaking a lot of URLs

[GuillaumeGomez]

I'll add it.

[scottmcm]

It seems really unfortunate that this would make even incredibly common things like struct.Vec.html have URLs that are materially worse (in my opinion).

Could it at least use a slightly more complicated scheme to take advantage of the usual rust naming conventions? Like have struct vec be the one that gets struct.-vec.html, since it's normal for structs to start with an upper-case letter, and let struct Vec continue to be struct.Vec.html?

[Manishearth]

I agree, I don't think it's good that common types get affected by this

[Nemo157]

That wouldn't help with cases where the case differs mid-identifier and could be valid either way struct FooBar; struct FoObar;.

[GuillaumeGomez]

I don't understand your point @Nemo157. It would be -fo-obar. It would prevent name conflict, which is the primary issue here.

[Nemo157]

I was referring to @scottmcm's alternative.

[GuillaumeGomez]

Oh my bad. Then yes, I completely agree with you. :)

[scottmcm]

@Nemo157 Yes, that's not a complete fix, as it'd still change struct.FooBar.html to struct.Foo-Bar.html or similar.

But I don't think we should break non-agglutinative names just because we have to break agglutinative ones.

[GuillaumeGomez]

No, I guess my explanations are really bad... All capital letters are replaced by a dash followed by the lowercase version. So FooBar will become struct.-foo-bar.html. With this, no more conflicts.

[scottmcm]

Could you provide more details about those places?

My first instinct here is that having two structs in the same module that differ only by case is something that people should never be doing, and thus this should never come up. TBH I'd be inclined to make a deny-by-default lint for it, and say that if you do it and the docs don't work I don't really care. But maybe there's an important situation I'm missing, so having that called out here could easily change my mind about it.

The places where things differ in only case that don't bother me are things like fn.default.html and trait.Default.html, where the item-kind disambiguator is already making the URL unique.

(Then we could special-case keyword.*.html to make it work. Or just make one doc for both of them.)

[Manishearth]

Yeah this seems like a major change for very very little benefit in a case people should not be doing at all.

[GuillaumeGomez]

But as soon as we enter into the "binding zone", it's just conflicts everywhere. What to do then? Rename items? That could work, but that doesn't sound like a good idea... This is one of the oldest rustdoc issue for a good reason after all. I think it's a really good change because it keeps a recognizable URL scheme while preventing name conflicts. But well, debates are what RFCs are for after all.

[GuillaumeGomez]

Also, special case things is rarely a good idea.

[Manishearth]

@GuillaumeGomez what do you mean by the binding zone?

I do not think using dashes randomly is a recognizeable URL scheme.

[GuillaumeGomez]

It's not random (If the explanation in the RFC isn't good, please tell meso I can update it!).

It replaces all capitalized letters by adding a dash before them and then putting the non-capital letter behind. So hELoPeople becomes h-e-lo-people.

For the "binding zone", I'll take a const example:

/usr/include/openssl/crypto.h
49:#  define SSLeay_version          OpenSSL_version
51:#  define SSLEAY_VERSION          OPENSSL_VERSION

[Manishearth]

Yes, by "randomly" I meant "not in a way that is intuitively predictable"

[scottmcm]

Also, special case things is rarely a good idea.

While I agree in general, I see no issue with special-casing keyword, since it's already a magic special case anyway. It's not like users can add keyword yolo; items in their crates that we need to document.

[GuillaumeGomez]

They actually can. :p

The idea behind doc(keyword) is to allow proc-macro crates to add documentation for their own keywords if they introduce new ones.

[clarfonthey]

Perhaps a reasonable alternative might be to not allow multiple pages on rustdoc to differ by name exclusively by case (perhaps via some sort of compatibility lint) and then offer some attribute that lets you change the name that's used in the URL to circumvent this. Also as a lint, we can also turn it off when this level of compatibility isn't needed, but keep it on for stdlib.

For example, we could do something like #[doc(url = "SelfT")] on the Self keyword to differentiate it from self.

[GuillaumeGomez]

@clarfonthey There is no way I see where we can do that. :-/

[kinnison]

What I care about (as rustup maintainer) is:

1. Currently 1.50 (and presumably 1.51) docs from Linux cannot be installed on macos or Windows, causing problems with cross-compilation workflows unless the user knows to set the minimal profile.
2. So long as we don't do something this will continue

One alternative could be:

1. we add an attribute to support setting the 'doc name' for something
2. we make rustdoc refuse to run if there are case-insensitive clashes after applying those renames

If we instead go for a renaming approach such as that described in the original RFC here by Guillaume then

1. We MUST ensure that the mapping scheme is stable (i.e. doesn't change if a clash comes/goes)
2. We should minimise the effect on "normally" cased things if possible

However we go about this we MUST solve this as it's causing pain now that we generate keyword.self.html and keyword.Self.html and it's probably been hurting others (outside of the stdlib) for much much longer.

[ogoffart]

rustdoc could generate mangled version only in the unlikely event a conflict actually occurs for this name. When that happens, it could generate a "disambiguation" page for the un-mangled name that would have links to the different options. (similar to wikipedia's disambiguation page) and that does not require JS.

(As I mentioned https://internals.rust-lang.org/t/pre-rfc-stable-rustdoc-urls/13099/12 )

So for example, with self, there would be both a keyword.self-1.html and keyword.Self-2.html and then a keyword.self.html would be a disambiguation page that have links to both version (maybe even with a short summary of the doc).

[GuillaumeGomez]

@ogoffart I explained in the RFC why this approach doesn't work, I even linked my PR which took this approach. keyword.self.html URL works for both self and Self if you have a case-insensitive file system, but not on case-sensitive. So from this point, either you generate links to keyword.self.html to handle all URLs (but then you might overwrite the "real" file on case-insensitive file systems) or you keep it as is but then keyword.Self.html will be 404 on case-sensitive file systems.

It simply doesn't work.

Another important point: it requires JS to work, or we support non-JS browsing and I'd prefer if we kept it this way.

[ogoffart]

That PR doesn't take the approach i describe at all, as it still seem to keep the URL the same in case of conflict. The idea would be that rustdoc would not generate link to keyword.self.html, it would only generate links to one or the other versions.

One would, in fact, not need keyword.self.html at all. Just keyword.self-1.html and keyword.Self-2.html could in fact be enough. The only reason to keep it would be to keep old link working or if some other tool generate the URL without knowing there is a case conflict.

And this keyword.self.html would not need Javascript at all, it would just contains link (in text form) to both versions:
A bit like this (quite meta) page: https://en.wikipedia.org/wiki/Disambiguation_(disambiguation)

[GuillaumeGomez]

It seems I misunderstood your comment, my bad.

But then there is no coherence and no way to know which -N corresponds to which one. What happens if we decide to add sElf and SeLf later on? In short, we break the current URL scheme and we introduce potential incoherence. I really don't like this approach either.

[jyn514]

Perhaps a reasonable alternative might be to not allow multiple pages on rustdoc to differ by name exclusively by case (perhaps via some sort of compatibility lint) and then offer some attribute that lets you change the name that's used in the URL to circumvent this. Also as a lint, we can also turn it off when this level of compatibility isn't needed, but keep it on for stdlib.

For example, we could do something like #[doc(url = "SelfT")] on the Self keyword to differentiate it from self.

We're hoping to do this as a short-term workaround for the standard library itself (rust-lang/rust#83678) so there's less urgency to get some form of RFC in quickly. However this would be a pretty big burden for crates outside the standard library. It would need user action to decide what the new name should be for every item that has a conflict. For autogenerated crates like windows-rs or stm32 that would be non-trivial to fix.

It also expands doc(...) even more, which I'd like to avoid doing if possible, but that's not a very strong objection.

I do think it would be nice in the meantime to add an allow-by-default lint on names that overlap, but I don't think the long-term plan should be to have users fix it manually.

[jyn514]

However this would be a pretty big burden for crates outside the standard library. It would need user action to decide what the new name should be for every item that has a conflict. For autogenerated crates like windows-rs or stm32 that would be non-trivial to fix.

Another use case where this would be hard to fix: -sys crates, which don't control the names of the items they export.

[Nemo157]

If anyone wants to look into more detail on situations in which existing crates are hitting issues with this overlap I previously compiled a list of all overlapping filenames in docs.rs (using data from around March 2020 IIRC), this has ~700k overlapping pairs, out of a total of ~308M files, so ~0.45% affected.

[GuillaumeGomez]

I do think it would be nice in the meantime to add an allow-by-default lint on names that overlap, but I don't think the long-term plan should be to have users fix it manually.

I don't think it's a good idea either. It's not up to the users to go around rustdoc limitations. Just like you said, renaming conflicted items could be a huge burden for them. Lints are for best practices, and I'm pretty sure it's not something we want rustdoc users to do. ;)

[ogoffart]

But then there is no coherence and no way to know which -N corresponds to which one. What happens if we decide to add sElf and SeLf later on?

That's a good point. Perhaps the -N could be some kind of hash (for example, Self is one uppercase followed by three lowercase, so 0b100 = 8) This way it stays coherent.
Adding a symbol that generates a new conflict will change the URL of the other symbol, but that would be mitigated by the disambiguation page on the old URL.

[GuillaumeGomez]

Adding a symbol that generates a new conflict will change the URL of the other symbol, but that would be mitigated by the disambiguation page on the old URL.

That remains a breaking change in the URL scheme. Another issue with that is if a conflict appears afterward because you add a new item, it means the links to the original item won't work anymore.

Last element (for the rustdoc internals): that requires rustdoc to know in advance for the URL conflicts to be able to generate the correct URL for intra-doc links.

[Manishearth]

However we go about this we MUST solve this as it's causing pain now that we generate keyword.self.html and keyword.Self.html and it's probably been hurting others (outside of the stdlib) for much much longer.

I think we can special case this one, and we can do that without an RFC. It's already being patched up in rust-lang/rust#83678

---

Here's a proposal:

For outside the stdlib, we can have rustdoc detect situations with clashes, arbitrarily pick a numbering based on lexicographic sorting (struct.foo.html, struct.Foo-1.html, struct.FOO-2.html), and show a warning that it did this. We can also provide #[doc(filename)] to override the name used in the file so that these crates can get to having stable URLs. These folks are broken anyway so this does not break them more.

This is great: we do not break unbroken docs, but broken docs get "fixed" in a way that is suboptimal but allows installation, and provides a path to be fixed. Users can then make their own choices on whether they wish to continue having these clashes -- which lead to unstable URLs if anything new is added -- or fix with #[doc(filename)]

When fixing broken stuff it is good to as much as possible limit the impact to the broken stuff itself.

The current proposal has multiple downsides:

• It breaks basically half the doc links out there. Maintaining redirects would bloat docs
• It changes the scheme to be kinda weird and confusing and hard to type from memory
• It removes the ability to urlcomplete a lot of docs

It introduces a new encoding scheme, and that is not to be taken lightly. Browser URL bars for example do not know anything about "rustdoc's bespoke case encoding scheme" and will not apply any normalization to autocomplete.

My proposal here has this downside:

• For crates with clashes, their doc links will be somewhat unstable (but can be fixed!)

This is still better than breaking all the docs (where everyone gets unstable links), and it's better than the status quo of "you have no docs for one of these" for crates with clashes.

I am very much strongly against anything that changes the URL scheme for existing crates that do not have this problem, especially something which introduces a new encoding scheme.

[Manishearth]

Please actually list these other alternatives and why they require JS to be enabled.

[GuillaumeGomez]

Good point. I'll describe them a bit more and add yours.

[Manishearth]

@ogoffart I explained in the RFC why this approach doesn't work, I even linked my PR which took this approach

You really didn't explain in the RFC at all; it's really hard to follow the background here. Just a couple links to PRs is not enough.

It is entirely possible to do this without JS. Just generate two files, and link to them directly as needed. The disambiguation page need not be fancy JS, it can literally just list the two options. I don't understand why JS is necessary at all, and I've gone through that PR.

[Manishearth]

This is a huge huge issue and I'm surprised we're taking this so lightly. I link to rustdoc pages in my writing at least once a week and usually once a day, there are millions of links out there.

[GuillaumeGomez]

It can be linked using rust version: https://doc.rust-lang.org/1.48.0/std/index.html

Fixing a 6 years problem seems more important to me. The important thing will be to communicate about this a lot.

[GuillaumeGomez]

I think we can special case this one, and we can do that without an RFC. It's already being patched up in rust-lang/rust#83678

This is not a solution, this is a hack. And a horrible one at that to fix the current rustup issue until we reach a conclusion.

Here's a proposal:

For outside the stdlib, we can have rustdoc detect situations with clashes, arbitrarily pick a numbering based on lexicographic sorting, and show a warning that it did this. We can also provide #[doc(filename)] to override the name used in the file so that these crates can get to having stable URLs.

This is great: we do not break unbroken docs, but broken docs get "fixed" in a way that is suboptimal but allows installation. Users can then make their own choices on whether they wish to continue having these clashes -- which lead to unstable URLs if anything new is added -- or fix with #[doc(filename)]

But by doing this, we introduce incoherence because an item URL can changed because of some unrelated change in the code.

When fixing broken stuff it is good to as much as possible limit the impact to the broken stuff itself.

The current proposal has multiple downsides:
* It breaks basically half the doc links out there. Maintaining redirects would bloat docs
* It changes the scheme to be kinda weird and confusing and hard to type from memory
* It removes the ability to urlcomplete a lot of docs

I don't see how "hard to type from memory" can be true here. It requires to know how it works, but that's it. Then it's just back to how it was. Also, to be fair, it doesn't remove the ability to urlcomplete, it simply makes it look less good. But the advantage is that the URL scheme will then be permanent.

It introduces a new encoding scheme, and that is not to be taken lightly. Browser URL bars for example do not know anything about "rustdoc's bespoke case encoding scheme" and will not apply any normalization to autocomplete.

Which is why we're having this RFC.

My proposal here has this downside:

* For crates with clashes, their doc links will be somewhat unstable

It can also change URL if an another item name enter in conflict with another one, introducing incoherence.

This is still better than breaking all the docs (where everyone gets unstable links)

This is your opinion, I don't agree with you on this but again, this is why we have RFC. However, it's not "everyone gets unstable links", it's a new URL scheme, which unlike the one you proposed won't have incoherences.

I am very much strongly against anything that changes the URL scheme for existing crates that do not have this problem, especially something which introduces a new encoding scheme.

That is the biggest downside of this RFC, indeed. I think it's worth it.

It is entirely possible to do this without JS. Just generate two files, and link to them directly as needed. The disambiguation page need not be fancy JS, it can literally just list the two options. I don't understand why JS is necessary at all, and I've gone through that PR.

You mean containing both items in the disambiguation page? That would make it double the size and would require some way from the browser to know which one to display. Otherwise, I don't see how you can do it. Anoter problem then will be on case-sensitive FS: you need to make all URLs go to the disambiguation page, and for that you need link files (ln -s), link files which might overwrite the disambiguation file on case insensitive FS. This is why such solutions cannot work.

If we choose to simply not handle the original URLs and directly go to both correct files, it can work, but then we go back to the incoherence I strongly disagree with.

[Manishearth]

This is not a solution, this is a hack. And a horrible one at that to fix the current rustup issue until we reach a conclusion.

I mean, sure, whatever, it fixes rustup's immediate issues. I'm trying to remove the sense of urgency from this RFC.

But by doing this, we introduce incoherence because an item URL can changed because of some unrelated change in the code.

These users have already broken docs, this is already true for them.

My proposal does not automatically fix the instability, but we lint about it, provide a sensible default, and provide these users a path to not having this instability.

You mean containing both items in the disambiguation page? That would make it double the size and would require some way from the browser to know which one to display. Otherwise, I don't see how you can do it.

No, I mean the disambiguation page would just contain two links. That's it. You don't need to do anything fancy to load docs onto the page. There's absolutely no need to do that in a disambiguation page; it's a disambiguation page its purpose is to disambiguate, not to show you everything.

Anoter problem then will be on case-sensitive FS: you need to make all URLs go to the disambiguation page, and for that you need link files (ln -s), link files which might overwrite the disambiguation file on case insensitive FS. This is why such solutions cannot work.

We should simply not do this. We do not need to make all URLs go to the disambiguation page, just one of them. We don't even have to make a disambiguation page in the first place, the solution still works.

The whole point of my proposal is that we explicitly declare such clashes to be things we do not support well, and provide users with a route to avoiding such clashes via doc(filename). With this path we do not need to worry about making the clash case "perfect", we can simply make it work.

You seem to have latched on to a particular solution of this kind, noticed that it's infeasible, and declared that that is why such solutions do not work. Please don't; please actually engage with this solution as stated instead of assuming it's the same as previous proposals.

This is your opinion, I don't agree with you on this but again, this is why we have RFC. However, it's not "everyone gets unstable links", it's a new URL scheme, which unlike the one you proposed won't have incoherences.

We break half the links out there. This is not perpetually unstable, but it's unstable in that we break a ton of stuff. That's instability.

I don't see how "hard to type from memory" can be true here. It requires to know how it works, but that's it.

I do not think people will learn how this works

[GuillaumeGomez]

I mean, sure, whatever, it fixes rustup's immediate issues. I'm trying to remove the sense of urgency from this RFC.

I see. yes, there is no urgency. The goal here is to fix the URL issue definitely.

But by doing this, we introduce incoherence because an item URL can changed because of some unrelated change in the code.

These users have already broken docs, this is already true for them.

My proposal does not automatically fix the instability, but we lint about it, provide a sensible default, and provide these users a path to not having this instability.

So we would expect from rustdoc users to go around rustdoc limitations. This is not a great user experience...

You mean containing both items in the disambiguation page? That would make it double the size and would require some way from the browser to know which one to display. Otherwise, I don't see how you can do it.

No, I mean the disambiguation page would just contain two links. That's it. You don't need to do anything fancy to load docs onto the page. There's absolutely no need to do that in a disambiguation page; it's a disambiguation page its purpose is to disambiguate, not to show you everything.

Oh I see. But that wouldn't work either like I explained on top. All conflict URLs have to be covered, which wouldn't be the case on case sensitive FS.

Anoter problem then will be on case-sensitive FS: you need to make all URLs go to the disambiguation page, and for that you need link files (ln -s), link files which might overwrite the disambiguation file on case insensitive FS. This is why such solutions cannot work.

We should simply not do this. We do not need to make all URLs go to the disambiguation page, just one of them. We don't even have to make a disambiguation page in the first place, the solution still works.

The whole point of my proposal is that we explicitly declare such clashes to be things we do not support well, and provide users with a route to avoiding such clashes via doc(filename). With this path we do not need to worry about making the clash case "perfect", we can simply make it work.

Except it doesn't work. k.self and k.Self should both work (and they would on case insensitive FS), but that wouldn't work on case sensitive FS.

You seem to have latched on to a particular solution of this kind, noticed that it's infeasible, and declared that that is why such solutions do not work. Please don't; please actually engage with this solution as stated instead of assuming it's the same as previous proposals.

This is your opinion, I don't agree with you on this but again, this is why we have RFC. However, it's not "everyone gets unstable links", it's a new URL scheme, which unlike the one you proposed won't have incoherences.

We break half the links out there. This is not perpetually unstable, but it's unstable in that we break a ton of stuff. That's instability.

The rustdoc URL scheme isn't stable yet. We're not proposing this out of nowhere. We tried multiple times to fix this issue and nothing worked because we tried to change as little as possible the URL scheme. Unfortunately it's a problem.

I don't see how "hard to type from memory" can be true here. It requires to know how it works, but that's it.

I do not think people will learn how this works

That is a fair argument.

[Manishearth]

So we would expect from rustdoc users to go around rustdoc limitations. This is not a great user experience...

This is a limitation imposed on us by the filesystem, either we work around it (as we're doing so in this RFC), or we give the users some choice in the matter

Oh I see. But that wouldn't work either like I explained on top. All conflict URLs have to be covered, which wouldn't be the case on case sensitive FS.
Except it doesn't work. k.self and k.Self should both work (and they would on case insensitive FS), but that wouldn't work on case sensitive FS.

Because we lint about it we don't have to be perfect about how it's handled, just do a sensible default.

[GuillaumeGomez]

So we would expect from rustdoc users to go around rustdoc limitations. This is not a great user experience...

This is a limitation imposed on us by the filesystem, either we work around it (as we're doing so in this RFC), or we give the users some choice in the matter

No, it's a limitation we created by picking this URL scheme. The FS should have been taken into account before that.

Oh I see. But that wouldn't work either like I explained on top. All conflict URLs have to be covered, which wouldn't be the case on case sensitive FS.
Except it doesn't work. k.self and k.Self should both work (and they would on case insensitive FS), but that wouldn't work on case sensitive FS.

Because we lint about it we don't have to be perfect about how it's handled, just do a sensible default.

But then the issue isn't solved, you just hacked something on top of it which didn't solve the original issue.

[GuillaumeGomez]

Another solution based on a disambiguator was suggested. To make things more smooth, this would only be applied on case insensitive file systems. I wrote a comparison between the two propositions there if anyone is interested: https://gist.github.com/GuillaumeGomez/a9d5e0d92987599a7966f5363543f81f

[GuillaumeGomez]

Closing this RFC as for now, another solution would be preferred rather than change all URLs.