Support {prefix} and {lowerprefix} markers in config.json dl key

[drmikehenry]

Hello,

The crates.io-index Git repository uses a nice directory structure to keep individual directory sizes under control.

When mirroring crates.io, it's useful to store crate files in a similar directory structure for the same reasons.

Cargo provides "markers" for use in the dl key of the config.json file in crates.io-index to allow flexibility in mapping a crate's name and version into a URL for the crate. The marker {crate} is replaced by the crate's name, and the marker {version} is replaced with the crate's version. The default URL template is https://crates.io/api/v1/crates/{crate}/{version}/download.

Currently, if a mirror of crates.io stores crates in a directory structure similar to that of crates.io-index, it's up to the server to construct the directory name from the crate name. This eliminates trivial web servers and file: URLs from hosting such a tree of crates.

This pull requests adds two new markers for the dl key in config.json, allowing Cargo to supply the directory name as part of the URL. The marker {lowerprefix} is the same directory name used within crates.io-index; it is calculated from the crate name converted to lowercase. The marker {prefix} is similar, but it uses the crate name as-is (without case conversion), which is useful for supporting older versions of Cargo that lack these markers; for example, nginx rewrite rules can easily construct {prefix} but can't perform case-conversion to construct {lowerprefix}. These new markers will provide implementation flexibility and simplicity for crate mirror servers.

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[alexcrichton]

Thanks for the PR and seems reasonable to me! I'm not sure that {prefix} and {lowerprefix} are the best names here but we can always bikeshed. In any case this is some new stable surface area, so I'm going to canvas the team:

@rfcbot fcp merge

[rfcbot]

Team member @alexcrichton has proposed to merge this. The next step is review by the rest of the tagged team members:

• @Eh2406
• @alexcrichton
• @ehuss
• @joshtriplett

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[joshtriplett]

I've had several discussions with some of the folks involved with creating the original directory structure with these prefix subdirectories, and each time, they've expressed regret at the use of those directories; they're much less relevant on modern filesystems. That shouldn't stop us from supporting the directory structure we have; I'm only mentioning it because anyone creating a new registry should seriously evaluate the directory structure and figure out what works best, rather than assuming that "Rust's upstream registry did it so it's probably a good idea". 😄

That said, having support for this seems reasonable:
@rfcbot reviewed

[drmikehenry]

Thanks for considering this pull request. As always (for me, anyway :-)), naming is hard. I'm happy to change {prefix} and {lowerprefix} to whatever is preferred.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[ehuss]

A agree, the prefix/lowerprefix names are a little vague, but I don't have any better suggestions.

[ehuss]

This might be a little clearer to list each substitution separately, maybe something like this:

- `dl`: This is the URL for downloading crates listed in the index. The value
  may have the following markers which will be replaced with their
  corresponding value:

  - `{crate}`: The name of crate.
  - `{version}`: The crate version.
  - `{prefix}`: A directory prefix computed from the crate name. For example,
    a crate named `cargo` has a prefix of `ca/rg`. See below for details.
  - `{lowerprefix}`: Lowercase variant of `{prefix}`.

  If none of the markers are present, then the value
  `/{crate}/{version}/download` is appended to the end.

Alternatively, maybe the entire discussion of the dl format could be moved into a separate section. At first reading, it seemed a little confusing because there are two concepts (the index directory layout, and the download directory layout).

[drmikehenry]

Thanks; I've incorporated your wording improvements into the pull request.

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

The RFC will be merged soon.

[alexcrichton]

@bors: r+

[bors]

📌 Commit b375bea has been approved by alexcrichton

[bors]

⌛ Testing commit b375bea with merge 22a112b...

[bors]

☀️ Test successful - checks-azure
Approved by: alexcrichton
Pushing 22a112b to master...