Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: 'mod tidy' downgrades an unused indirect dependency upgraded by 'get -u' #29702

Closed
SamWhited opened this issue Jan 12, 2019 · 16 comments
Closed
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@SamWhited
Copy link
Member

SamWhited commented Jan 12, 2019

I thought I remembered an issue about this being fixed already, but I couldn't find it and it's still happening and causing problems. Sorry of this is noise and the issue is still floating around somewhere.

The specific dependency in this example that's being stripped out and re-added does not use modules, but does use semantic versioned tags which testify is pinning to.

What version of Go are you using (go version)?

$ go version
go version devel +a2bb68de4d1c 20190111160859 linux/amd64

Does this issue reproduce with the latest release?

It happens with go1.12beta2, go1.12beta1, and go1.11.4.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN="/home/sam/bin"
GOCACHE="/home/sam/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/sam/go"
GOPROXY=""
GORACE=""
GOROOT="/home/sam/Projects/go"
GOTMPDIR=""
GOTOOLDIR="/home/sam/Projects/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/sam/test/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build393255074=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Create a program that imports a package that exhibits this behavior. I noticed it with the Stripe SDK, and testify seems to be the dependency causing the problem so the following also has the same issue:

package main

import (
	"fmt"

	"github.com/stretchr/testify/assert"
)

func main() {
	_ = assert.CallerInfo
	fmt.Println("vim-go")
}

Now run go mod init testmod, and go get -u. You will end up with the following go.mod file:

module testmod

go 1.12

require (
	github.com/davecgh/go-spew v1.1.1 // indirect
	github.com/stretchr/objx v0.1.1 // indirect
	github.com/stretchr/testify v1.3.0
)

now run go mod tidy and you wind up with the following:

module testmod

go 1.12

require (
	github.com/davecgh/go-spew v1.1.1 // indirect
	github.com/stretchr/testify v1.3.0
)

What did you expect to see?

The file does not change.

What did you see instead?

Every time you run go mod tidy it strips out indirect dependencies inserted by go get -u and you end up with unmerged changes in your repo.

@mvdan mvdan added the modules label Jan 12, 2019
@mvdan mvdan changed the title go mod tidy and go get -u fight eachother cmd/go: mod tidy and get -u fight each other Jan 12, 2019
@mvdan mvdan added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 12, 2019
@mvdan
Copy link
Member

mvdan commented Jan 12, 2019

/cc @bcmills

@bcmills
Copy link
Contributor

bcmills commented Jan 14, 2019

Looks like they agree on the build list but disagree about whether dependencies are indirect:

$ go get -u
go: finding github.com/davecgh/go-spew v1.1.1
go: finding github.com/stretchr/objx v0.1.1
go: downloading github.com/davecgh/go-spew v1.1.1

$ cat go.mod
module golang.org/issue/scratch

require (
        github.com/davecgh/go-spew v1.1.1 // indirect
        github.com/stretchr/objx v0.1.1 // indirect
        github.com/stretchr/testify v1.3.0
)

$ go list -m all
golang.org/issue/scratch
github.com/davecgh/go-spew v1.1.1
github.com/pmezard/go-difflib v1.0.0
github.com/stretchr/objx v0.1.1
github.com/stretchr/testify v1.3.0

$ go mod tidy

$ cat go.mod
module golang.org/issue/scratch

require (
        github.com/davecgh/go-spew v1.1.1 // indirect
        github.com/stretchr/testify v1.3.0
)

$ go list -m all
golang.org/issue/scratch
github.com/davecgh/go-spew v1.1.1
github.com/pmezard/go-difflib v1.0.0
github.com/stretchr/objx v0.1.0
github.com/stretchr/testify v1.3.0

@bcmills bcmills added this to the Go1.13 milestone Jan 14, 2019
@bcmills
Copy link
Contributor

bcmills commented Jan 14, 2019

Wait, no. go mod tidy ends up downgrading; that might be a bug.

@bcmills
Copy link
Contributor

bcmills commented Jan 14, 2019

This case is interesting, though: nothing in the build actually imports from objx, and nothing in objx affects the versions of other libraries, so perhaps go mod tidy is correct to remove it from the go.mod file even though it remains in the transitive module graph.

@SamWhited
Copy link
Member Author

SamWhited commented Jan 14, 2019

This case is interesting, though: nothing in the build actually imports from objx.

The tests do though (via testify/mock), and it appears that mod tidy is adding things behind build tags and test files generally; it's just this one that it's stripping out for some reason.
Even though it doesn't affect any other libraries, if we require pinning to a specific version of objx stripping it out might mean that we pull in a newer version that's broken, no?

@bcmills
Copy link
Contributor

bcmills commented Jan 14, 2019

The tests do though (via testify/mock),

Nothing in the import graph imports testify/mock. go mod tidy only considers the packages transitively imported by the main module and the transitive dependencies of the tests of those packages.

if we require pinning to a specific version of objx stripping it out might mean that we pull in a newer version that's broken, no?

In this case, go test all literally doesn't build anything in objx, so the only way it could be broken is through some invalid module requirement.

@SamWhited
Copy link
Member Author

Nothing in the import graph imports testify/mock. go mod tidy only considers the packages transitively imported by the main module and the transitive dependencies of the tests of those packages.

doc.go imports testify/http which imports testify/mock in a test_round_tripper.go, unless I'm still misunderstanding. It's a weird _ import though. I don't really understand why, not having ever used testify except transitively. It doesn't appear to have any import side effects (thank goodness).

In this case, go test all literally doesn't build anything in objx, so the only way it could be broken is through some invalid module requirement.

I see; I suppose if it's an _ import and nothing is actually used that's fair.

@bcmills
Copy link
Contributor

bcmills commented Jan 14, 2019

doc.go imports testify/http which imports testify/mock

Nothing in the example program imports github.com/stretchr/testify (that is, the package containing doc.go) itself.

@SamWhited
Copy link
Member Author

oops, sorry, you're right of course.

@bcmills bcmills changed the title cmd/go: mod tidy and get -u fight each other cmd/go: 'mod tidy' downgrades an unused indirect dependency upgraded by 'get -u' Jan 14, 2019
@rsc
Copy link
Contributor

rsc commented Jan 24, 2019

Why did go get -u decide to look at objx?

@thanasik
Copy link

thanasik commented Apr 3, 2019

Having this issue as well, but with local private dependencies. Both go mod tidy and go mod vendor alter my go mod file this way, and running go build on the modified go.mod file results in an error during build saying cannot find module for path (though it does actually vendor it, the files are in the vendor dir). I have to manually add the dependency back in order to get the build working. For context, my go version is go version go1.12.1 linux/amd64

@bcmills
Copy link
Contributor

bcmills commented Apr 9, 2019

This is probably a side-effect of #26902.

@thepudds
Copy link
Contributor

thepudds commented May 5, 2019

This is probably a side-effect of #26902.

Now that #26902 is resolved, I tried to reproduce this using tip (e.g., go get golang.org/dl/gotip && gotip download).

As hoped or expected, github.com/stretchr/objx now no longer shows up after the initial go get -u, and then running go mod tidy has no impact:

gotip mod init testmod
cat <<EOF > main.go
package main
import (
    "fmt"
    "github.com/stretchr/testify/assert"
)
func main() {
    _ = assert.CallerInfo
    fmt.Println("vim-go")
}
EOF
gotip get -u
cp go.mod go.mod_orig
gotip mod tidy
diff go.mod_orig go.mod
cat go.mod

which results in no diff reported, and go.mod contains:

module testmod

go 1.13

require (
        github.com/davecgh/go-spew v1.1.1 // indirect
        github.com/stretchr/testify v1.3.0
)

I used go version devel +ba9bc8e5 Sun May 5 12:55:21 2019 +0000.

Also, gotip get -u all seems to give same result as gotip get -u (which seems expected as well).

I did not however attempt to evaluate all the commentary in this issue -- I only tried the first set of steps reported here (in comment #29702 (comment)).

@SamWhited would you be willing to try this with tip? Does this seem resolved from your perspective, or does it seem there are still pieces pending?

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 6, 2019
@gopherbot
Copy link
Contributor

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@colega
Copy link
Contributor

colega commented Jul 2, 2019

This still happens on go version go1.12.6 darwin/amd64 and go version devel +d410642 Mon Jul 1 21:30:23 2019 +0000 darwin/amd64 (gotip).

We're reproducing it in the following scenario, consider two libraries github.com/zero-nine-three and github.com/one-zero-zero that import nothing but github.com/prometheus/client_golang on versions v0.9.3 and v1.0.0 respectively:

$ tail -n +1 ../zero-nine-three/go.mod ../one-zero-zero/go.mod
==> ../zero-nine-three/go.mod <==
module github.com/zero-nine-three

go 1.12

require github.com/prometheus/client_golang v0.9.3

==> ../one-zero-zero/go.mod <==
module github.com/one-zero-zero

go 1.12

require github.com/prometheus/client_golang v1.0.0

Then go mod tidy removes and go get adds the github.com/go-logfmt/logfmt v0.4.0 // indirect dependency:

$ go mod tidy

$ cat go.mod
module bug

go 1.12

replace github.com/one-zero-zero => ../one-zero-zero

replace github.com/zero-nine-three => ../zero-nine-three

require (
	github.com/one-zero-zero v0.0.0-00010101000000-000000000000
	github.com/zero-nine-three v0.0.0-00010101000000-000000000000
)

$ go get

$ cat go.mod
module bug

go 1.12

replace github.com/one-zero-zero => ../one-zero-zero

replace github.com/zero-nine-three => ../zero-nine-three

require (
	github.com/go-logfmt/logfmt v0.4.0 // indirect
	github.com/one-zero-zero v0.0.0-00010101000000-000000000000
	github.com/zero-nine-three v0.0.0-00010101000000-000000000000
)

We have noted that this behavior is not reproduced if we're importing a library without a hostname (i.e., zero-nine-three instead of github.com/zero-nine-three)

This behavior forces us to disable the tidyness check in our CI pipelines, since our tests are adding the indirect dependency.

@bcmills PTAL and ask for more info if needed.

P.D.: I've omitted the .go files in the example containing just underscore-imports to make tidy keep the imports, and the go.sum files.

@bcmills
Copy link
Contributor

bcmills commented Jul 11, 2019

@colega, this was fixed as #26902 and will not be backported to the 1.12 branch. If you can still reproduce it (or something similar) using go1.13beta1, please open a new issue.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

8 participants