WIP: implement vendoring for git dependencies

[bennofs]

This is a proof-of-concept for vendoring git dependencies with cargo-vendor using rust-lang/cargo#3992.

Open questions:

1. To download git dependencies, I re-used the code from cargo for downloading packages. For the --host option, I think we need to supply a registry replacement to cargo to make it work again. This raises the question: should we just allow the user to provide a custom .cargo/config for vendoring? That would be more flexible than the current --host option, as it would also allow specifying a different git source.

2. For generating the .cargo/config template, we need to generate a source config for all sources that we intend to override. Should the code to turn SourceId into a Toml config be part of cargo perhaps? We would need a way to find out if a source is a local-registry or a registry anyway (which afaik is not currently possible) so there is cargo change necessary in any case.

3. Right now, all sources get replaced with a single directory sources. Is this ok, or should there be a separate directory for each source that we want to replace?

[bennofs]

This also still needs some tests of course

[alexcrichton]

Hey sorry I haven't forgotten this! I think I'll run out of time this week to take a look but I'll try to get to this next week.

[Mic92]

If your pull request gets merged and #48
we can start replacing our registry hacks in nixpkgs

[sunshowers]

Any chance this could move forward? We have some manual hacks in place at the moment, and having a comprehensive solution in cargo vendor would be great.

[alexcrichton]

@Sid0 yeah this still needs upstream support from Cargo itself, I just need to get around to rebasing that and figuring out whether this is the right way to approach these problems.

[alexcrichton]

Ok I've rebased this to master and I've touched up the support here and there with the changes that have happened over the past few months. I think this is ready to go now.

@bennofs can you double check and make sure I didn't break anything you intended? Otherwise I'll likely r+ rust-lang/cargo#3992 soon and then merge this once that's in nightlies.

[bennofs]

@alexcrichton thanks! it's been a long time since I last looked at this, and I don't have a large testcase so not sure if I catch it if it breaks in some edge case.

[Mic92]

Does this also store the repository (.git) of the crate?

[Mic92]

@bennofs did you planned to gives this a try for nixpkgs? I could replace our current (hacky) approach.

[alexcrichton]

No, it should not

[Mic92]

Good to hear. .git is not reproducible by default.