Don't vendor dependencies

[dlsniper]

Hi,

While it's a good practice to vendor dependencies, in case of libraries, this can result in headaches.
In this case, it would be ok to have a file with the revision / version of the libraries but let the users pull the dependencies manually / via a tool of their preference.
I know there's no agreed format just yet for Go but maybe this would also help the discussion around it and push things forward.
Thank you.

[jurij]

Related Use Case:

jkmb:server juRiii$ godep save ./...
godep: Package (github.com/jmespath/go-jmespath) not found
jkmb:server juRiii$ grep -R --include="*.go" "github.com/jmespath/go-jmespath" /Users/juRiii/dev/go/src/
/Users/juRiii/dev/go/src//github.com/aws/aws-sdk-go/aws/awsutil/path_value.go:    "github.com/jmespath/go-jmespath"
/Users/juRiii/dev/go/src//github.com/aws/aws-sdk-go/vendor/github.com/jmespath/go-jmespath/fuzz/jmespath.go:import "github.com/jmespath/go-jmespath"
jkmb:Godeps juRiii$ go version
go version go1.6 darwin/amd64

[StabbyCutyou]

Not sure how widely supported it is, but the author of govendor has been maintaining a spec for dependency files in hopes to get it adopted by more of the tools.

Something to consider https://github.com/kardianos/vendor-spec

[jasdel]

Thanks for contacting us. I'd like to learn more about the issues you are seeing with the SDK vendoring its dependancies?

If there were a generic pattern the SDK could follow supported by the base Go tooling it would be great to investigate how the SDK could use this. Regrettably there is no champion pattern accepted by the community or Go tooling. This is why the SDK uses vendoring to ensure the upstream dependancies needed by the SDK will not cause users to break if the upstream dependancies makes a breaking change. Which has happened.

[dlsniper]

Hi, thanks for the awesome fast reply.

In the case of @juriii the problem is that even if this project vendors the code, the tool he's using is not aware of the the fact that the library already has the dependencies vendored and it tries to copy them from GOPATH (which will fail since they are not there, would be interesting to know how he go getted the package, by doing go get aws-sdk-go or go get aws-sdk-go/...).

Maybe a simple fix could be to tell the users to do the later, with appending /... to the package when go getting it (needs to be tested) (cc @juriii, see this).

On the large side of the issues, this is still pretty much a yet to be solved problem in Go but the general consensus is to not vendor the dependencies in case the project is a library and not an application. Fortunately enough, you are doing the nice thing and not also rewriting the import path (which a bigger problem). Of course, you could use a tool which has a file in which it saves the revision for each dependency and have only that added to the repository (but that means a big transition and forcing users to use that specific tool + the tool should be aware of the vendor/aws/vendor revisions)

This should also be asked / discussed on the mailing list / with the Go team as we can either have a community driven solution, like @StabbyCutyou mentioned vendor-spec, or we can have the Go team creating a standard for this.

[jurij]

@dlsniper I have used go get -u github.com/aws/aws-sdk-go because of go1.6. I have not thought about that vendoring feature of any dependency I am using.

@jasdel Since most users are using this as a Library and not contributing, at least an Information in the README.md would be helpful.

[jasdel]

Thanks for your feedback, and suggestion of tools. We're taking a look at the options, but currently we don't think there is a good solution thats widely accepted by the Go community. With multiple competing solutions like Glide, Godep, and multiple vendor spec formats, we think its too soon to choose a specific format and be stuck with it. The /vendor folder is supported by all Go 1.5 with 1.5 vendoring experiment and +1.6 out of the box.

The downside of go get -u github.com/aws/aws-sdk-go/... is that users would be downloading several more dependancies than those needed to build applications against the SDK. Specifically, the testing libraries used by the SDK. Along with the golang.org/x/tools dependancies used by the deprecated awsmigration tool.

Instead of requiring users of the SDK to use a non-standard tool, we'd prefer to provide the most generic support possible, while also providing dependency versioning locking. With the vendoring folder we can guarantee that the SDK will work out of the box when downloaded without worrying about vendoring conflicts.

In addition, the SDK's README.md can use some updates to include more detailed information how retrieve the SDK, and what the different methods of retrieval will entail.

[dlsniper]

@jasdel I completely agree with your points.
I would suggest raising the issue with the Go team on the mailing list https://groups.google.com/d/forum/golang-dev to see what's their opinion on this as come Go 1.7 the current solution will be here to stay (for better or worse) and feedback like this could help them out to understand the issue and either have them or the community come to an agreement on what needs to be done moving forward. I've raised a issue as well, unrelated to this, and there are several other threads with different problems as well, most recently (and somewhat similar) https://groups.google.com/forum/#!topic/golang-dev/4FfTBfN2YaI

Thank you!

[e-dard]

@jasdel

The downside of go get -u github.com/aws/aws-sdk-go/... is that users would be downloading several more dependancies than those needed to build applications against the SDK. Specifically, the testing libraries used by the SDK.

That shouldn't be the case. As far as I'm aware go get has never fetched test dependencies; you have to explicitly provide the -t flag to go get to fetch those.

[jasdel]

@e-dard I verified that retrieving the SDK with go get -u github.com/aws/aws-sdk-go/... will still download some of the testing libraries because they are imported from utility packages within the SDK's repository. Specifically the awstesting package.

With that said the simplest way to retrieve the SDK with only decencies needed for runtime is: go get -u github.com/aws/aws-sdk-go/service/...

[jasdel]

Thanks for the feedback, I've updated the README.md to be clearer on what to expect when retrieving the SDK.

[jasdel]

I created a PR #1544 which adds the dep metadata files to the SDK. This will allow applications to correctly manage the SDK's dependencies in the context of the application.

I do not think we can safely remove the vendor folder from the SDK without breaking users that do not vendor, and not using dep. In a future version of the SDK a vendor folder would not be needed.

[roytanmoy]

Is this issue resolved? I am still getting the similar error with Go version 1.11.2.
../aws/aws-sdk-go/aws/awsutil/path_value.go:9:2: cannot find package "github.com/jmespath/go-jmespath"

[jasdel]

@roytanmoy Are you using the latest version of the SDK? v1.16.2 updated the SDK's dependency on go-jmespath to be the latest version.