Proposal: Resolve In Glide Cache Instead of Vendor

[mattfarina]

Currently, Glide fetches packages (as repos) into the vendor/ directory and then resolves the dependency tree. As new packages are needed they are fetched and set to the needed version.

This proposal is to break this process apart. Packages would be fetched and versions resolved in the Glide cache and then exported (just the versions source without vcs information) to the vendor/ directory.

Pros:

• When packages are vendored, that is stored in your apps VCS, there will be a major performance improvement when a warm cache.
• Sometimes resolution has errors which causes Glide to exit before completion. This can leave the vendor/ directory in a bad state that needs to be fixed. This would no longer happen as only when Glide finds a new good state would it put source in vendor/.
• Some flags such as --cache, --strip-vcs, --update-vendored would all no longer be needed.
• Confusion around subpackages in Git would go away.

Cons:

• If you checkout a dependency into the vendor/ directory, make changes to it from there, and push those changes you wouldn't be able to do that while Glide was working on your tree.
• If you are in development and using Docker and you want them to share a cache you may need to pass in the shared location to Glide for the cache.
• Glide would leverage a global cache lock which would cause parallel glide operations (processes) to take longer on the same system.

There are likely more pros and cons. If you know of any please note them in the comments.

Thoughts?

[chancez]

This is what i would prefer. I think the global cache lock is a good idea, and most people who want multiple glide processes can have multiple cache directories (each with their own lock). It seems like it would also make it easier to support updating individual vendored packages at a time.

[sdboyer]

Yep, big +1.

Also, this is what vsolver does already. If you're seriously considering merging in vsolver anytime soon, you might want to hold off on doing this work, as it'll be redundant.

That doesn't necessarily mean #384 itself per se, as that's about using the solving engine for making actual decisions. It could just mean merging in vsolver in order to use the SourceManager as a workhorse. That would probably entail other changes, too, but it's still a smaller target. (we can also get sdboyer/gps#27 squared away, too)

[mattfarina]

@sdboyer I'm not sure when vsolver will be merged in (mostly because I'm behind on it). This is a major change no matter when or how it happens and I wanted to open it up for discussion.

[sdboyer]

@mattfarina sure, makes sense. It's near enough to MVP now that I do have docs near the top of my TODO list. Those should help.

Work towards a central cache might also be an opportunity to catch up with the piece that's more easily grokked (the SourceManager), which could make the rest of the grok less daunting. It'd also make the transition itself less significant for glide in general.

[dmitris]

would it be possible to customize the cache location? It would be great if you could pass as a parameter an alternative locaion such as a different GOPATH where you already have the correct origin/version of the dependencies, so that they would not have to be fetched over networked. For us it would greatly speed up the builds.

[sdboyer]

@dmitris you can set the location of the cache, though somewhat indirectly - you set the location of glide's "home", and the cache is located under there. So, no, you can't set it to an existing GOPATH.

There are two separate things in your question, though, so lemme address them separately

such as a different GOPATH where you already have the correct origin/version of the dependencies

In the general case, the only safe assumption we can make is that GOPATH is an unkempt mess of versions, specifically because GOPATH tends to have been populated by go get, the honey badger of versioning. Inferring the user's intended version from them is the source of a lot of the problems that drove the creation of something like glide in the first place.

The good news there is that the init process is generally responsible for pulling the information from your GOPATH into the glide.yaml manifest - so that information can still be used. But it's not really appropriate to use at solve-time, which is more what this issue is about.

so that they would not have to be fetched over networked. For us it would greatly speed up the builds.

This is much more doable, and is in part what the --use-gopath opt is intended to accomplish. However, that only helps for the specific situation where there's a repo on your GOPATH that's not in the cache yet - which, the more you use glide, becomes an increasingly uncommon situation. So, while copying the repos from the GOPATH to the cache could be a bit of a one-time speedup, it sounds like you're thinking of across-the-board gains. I'm not sure how that would happen; maybe you're imagining that this central cache would do something it doesn't?

[mattfarina]

@dmitris A GOPATH is hard to use as a cache for a couple of reasons (that I'm happy to discuss and find better solutions for).

1. Forks. If you have a project at github.com/example/foo and a fork at github.com/me/foo that you want used for github.com/example/foo how do you consistently manage that? Without some outside application mucking with things in a way you can't see or work with? Dealing with alternative locations, sometimes multiple ones, is a use case for Glide and complicates things.
2. Access controls. https://example.com/foo/bar.git can be different from git@example.com:foo/bar.git. Some hosts allow different users to have see different branches and tags. I'm aware of multiple systems used in on-prem implementations that do this. To respect access we need to treat those differently.

The general open source case where you get it from the root source is easy. Managing the complicated cases get's harder.

But, using the cache (which can be hot and update only the missing bit) can allow for some real speed improvements. It's in place but turned off because the copy from cache element is too slow. It's implemented in a cross platform way (to support things like windows). For many the network is actually faster.

Part of this end up making the cache operations much faster. For example, if a commit being installed was already in the cache there's no reason to update. Exporting (using the vcs optimized tools) is amazingly fast and can be used.

[mattfarina]

This was just merged.

[chr0n1x]

@mattfarina not sure how as I have not looked at the code for the implementation, but this is causing some issues with my docker (~1.1x) containers/environments.

I have a workflow where I directly mount my local repository into a golang workspace within a docker container. This is so that I can restart running programs by just doing a docker restart after changing small bits of code. So something the equivalent of:

docker run -ti --rm \
    -v $(pwd):/gopath/src/github.com/myorg/myrepo \
    -w /gopath/src/github.com/myorg/myrepo \
    golang \
    bash -c "go get github.com/Masterminds/glide && glide install && bash"

The issue is that glide install ends up failing with:

[ERROR]    Unable to export dependencies to vendor directory: rename /tmp/glide-vendor175037764/vendor /gopath/src/github.com/myorg/myrepo/vendor: invalid cross-device link

A temporary workaround that I've found was something like:

docker run -ti --rm \
    -v $(pwd):/mount/myrepo \
    -w /gopath/src/github.com/myorg/myrepo \
    golang \
    bash -c "go get github.com/Masterminds/glide && glide install && cp -R /mount/myrepo /gopath/src/github.com/myorg && cd. && bash"

Which is very sad.

Is this a known issue? Any ideas on how to deal with this in a saner fashion? Wont fix? Or will fix? 😄

[chr0n1x]

Oh! I just saw #554 - which I believe is the same issue. Sorry for the spam, thanks for reading :)