x/tools/gopls: when working in vendor mode autoimport adds packages from vendor dir

[inliquid]

gopls version

>gopls -v version
Build info
----------
golang.org/x/tools/gopls master
    golang.org/x/tools/gopls@(devel)
    github.com/BurntSushi/toml@v1.2.0 h1:Rt8g24XnyGTyglgET/PRUNlrUeu9F5L+7FilkXfZgs0=
    github.com/google/go-cmp@v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
    github.com/sergi/go-diff@v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
    golang.org/x/exp@v0.0.0-20220722155223-a9213eeb770e h1:+WEEuIdZHnUeJJmEUjyYC2gfUMj69yZXw17EnHg/otA=
    golang.org/x/exp/typeparams@v0.0.0-20220722155223-a9213eeb770e h1:7Xs2YCOpMlNqSQSmrrnhlzBXIE/bpMecZplbLePTJvE=
    golang.org/x/mod@v0.6.0-dev.0.20220419223038-86c51ed26bb4 h1:6zppjxzCulZykYSLyVDYbneBfbaBIQPYMevg0bEwv2s=
    golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4 h1:uVc8UZUe6tr40fFVnUP5Oj+veunVezqYl9z7DYw9xzw=
    golang.org/x/sys@v0.0.0-20220808155132-1c4a2a72c664 h1:v1W7bwXHsnLLloWYTVEdvGvA7BHMeBYsPcF0GLDxIRs=
    golang.org/x/text@v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
    golang.org/x/tools@v0.1.13-0.20220928184430-f80e98464e27 => ../
    golang.org/x/vuln@v0.0.0-20221010193109-563322be2ea9 h1:KaYZQUtEEaV8aVADIHAuYBTjo77aUcCvC7KTGKM3J1I=
    honnef.co/go/tools@v0.3.3 h1:oDx7VAwstgpYpb3wv0oxiZlxY+foCpRAwY7Vk6XpAgA=
    mvdan.cc/gofumpt@v0.3.1 h1:avhhrOmv0IuvQVK7fvwV91oFSGAk5/6Po8GXTzICeu8=
    mvdan.cc/xurls/v2@v2.4.0 h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc=
go: go1.19.2

go env

>go env
set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\***\AppData\Local\go-build
set GOENV=C:\Users\***\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\***\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\***\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.19.2
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=NUL
set GOWORK=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\***\AppData\Local\Temp\go-build1332515651=/tmp/go-build -gno-record-gcc-switches

What did you do?

When working in vendor module mode, and file does not contain imports yet, press Ctrl-S to force format and auto import actions.

What did you expect to see?

Imports added with correct package names, for example: google.golang.org/grpc

What did you see instead?

Packages added from vendor dir, for example: "path/to/my/module/vendor/google.golang.org/grpc"

[findleyr]

Thank you for the report.

CC @adonovan, as this coincidentally seems very similar to https://go.dev/cl/443636: clearly a package path and import path are being confused here (though I don't necessarily think this will be fixed by that CL -- goimports is mostly disconnected from the rest of gopls).

@inliquid are you by any chance working on an open-source project? If not, we can try to reproduce from scratch.

[inliquid]

@findleyr, unfortunately this is a private repo.

[inliquid]

I also found that when you use "References" or "Implementations" features, VS Code also displays paths from vendor:

[jstroem]

Working on a private repo I've noticed the same.

[gopherbot]

Change https://go.dev/cl/448377 mentions this issue: gopls/internal/lsp/cache: use Package.FileSet where possible

[adonovan]

Hi @inliquid, I've tried to reproduce this bug a number of ways but with no success. (For example, I vendored a number of packages, created a file with missing imports of packages among the vendored set, and ran 'gopls imports' on them, and each time it got the right answer. I tried various versions of gopls before recent changes in relevant logic. And I tried it without and without a go.work file. I also looked at the various places in our code that we've flagged as suspicious in the way in which they convert between import paths and package paths, and none of them looks like an obvious candidate to explain it.)

I understand this is private code you can't share, but next time you encounter the problem, would you be willing to spend a few minutes trying to create a reproducible test case in terms of packages you can share? As soon as I can reproduce this I can work on a fix. Thanks.

[inliquid]

@adonovan I work with gopls@master and update it regularly, does it say anything?

Just tried once again by adding glob.Glob in one of working files and pressing Ctrl-S:

If you have any idea what additional information I can provide please let me know. You are correct this is private repo so I can't share its code unfortunately.

[adonovan]

Still no luck reproducing this.

The "implementations" issue you mention in #56291 (comment) is likely a dup of #56169, which should now be fixed.

[inliquid]

Well the problem is definitely still present. Except for sharing the project files, what else can I do to help in solving it?

[bferullo]

I, and several of my coworkers, are also experiencing this issue. I hacked together a shell script that seems to reliably reproduce it (tested in git-bash and cygwin).

Requires gopls to be installed as a command (I ran go install golang.org/x/tools/gopls@latest).

#!/bin/sh

# create a new dir to hold our package
mkdir vendorbug
cd vendorbug

# create a simple main.go that uses github.com/google/uuid
cat > main.go <<EOT
package main

import (
	"fmt"

	"github.com/google/uuid"
)

func main() {
	fmt.Printf("This is a test package.")

	myUUID, _ := uuid.NewRandom()
	fmt.Printf("The UUID generated was: %s", myUUID)
}
EOT

# create a module
go mod init github.com/bferullo/vendorbug
go mod tidy
go mod vendor

# remove github.com/google/uuid import from main.go
sed --in-place -E '/github.com\/google\/uuid/d' ./main.go

# (verify here that main.go no longer has that import, if you want)

# run gopls imports
gopls imports -d main.go

# EXPECTED RESULT: "github.com/google/uuid" reappears in import block
# OBSERVED RESULT: "github.com/bferullo/vendorbug/vendor/github.com/google/uuid" appears in import block

[findleyr]

Huh, FWIW that script didn't reproduce the bug for me. I completely believe this bug exists, but it must depend on state (likely the state of the module cache).

[adonovan]

Same here. github.com/google/uuid appears each time. I tried with GOMODCACHE=$(mktemp -d) too. Could you @bferullo try with that env var and report whether it still happens?

[bferullo]

Same here. github.com/google/uuid appears each time. I tried with GOMODCACHE=$(mktemp -d) too. Could you @bferullo try with that env var and report whether it still happens?

It does still repro. I also imagine the issue has to do with some persistent state, but I can't figure it out either -- I even installed go, gopls, and gitbash on a machine that previously had none of those, and was able to repro it there as well.

[seankhliao]

can you give the exact versions of go and gopls you used?

[bferullo]

$ go version
go version go1.19.4 windows/amd64
$ gopls version
golang.org/x/tools/gopls v0.11.0
    golang.org/x/tools/gopls@v0.11.0 h1:/nvKHdTtePQmrv9XN3gIUN9MOdUrKzO/dcqgbG6x8EY=
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\bferu\AppData\Local\go-build
set GOENV=C:\Users\bferu\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\bferu\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\bferu\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.19.4
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=NUL
set GOWORK=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\bferu\AppData\Local\Temp\go-build375371210=/tmp/go-build -gno-record-gcc-switches

[inliquid]

I also can reproduce the issue with the above script. Since I also run it on Windows, can OS be a problem?

[bferullo]

Anecdotally, some colleagues who run OSX also encounter this issue. I'll see if I can get them to test the script specifically.

[adonovan]

Still no luck reproducing this on macOS using go1.19 (or go1.19.5) and the script and environment above, and either gopls@v0.11.0 or today's gopls@latest.

[inliquid]

I still can reproduce this with the script from above comments on a brand new Windows 11 machine (old one was Windows 10):

 import (
        "fmt"

+       "github.com/bferullo/vendorbug/vendor/github.com/google/uuid"
 )

PS: I'm using gopls@master with latest commits.

[gopherbot]

Change https://go.dev/cl/498695 mentions this issue: gopls/internal/regtest: add a test for goimports and windows vendoring

[findleyr]

The test in the above CL (based on the script provided by @bferullo reproduces the failure on our windows trybot. I'll fix this for gopls@v0.12.1 (which we expect not long after v0.12.0...)