cmd/go: toolchain directive in go.mod being updated unnecessarily

[dprotaso]

There doesn't seem to be a way to stop the toolchain directive from being updated when running various go commands

In our OSS project we rely on the go directive in the go.mod to ensure a min go version. We do not want to use the toolchain directive as we always expect it to match the go directive.

some user feedback

• https://gophers.slack.com/archives/C0VPK4Z5E/p1704471476388119
• https://gophers.slack.com/archives/C9BMAAFFB/p1708525315637839

Go Version

go version go1.22.0 darwin/amd64

Go ENV

expand

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/Users/dprotasowski/Library/Caches/go-build'
GOENV='/Users/dprotasowski/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/dprotasowski/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/dprotasowski/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/Cellar/go/1.22.0/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/Cellar/go/1.22.0/libexec/pkg/tool/darwin_amd64'
GOVCS=''
GOVERSION='go1.22.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/Users/dprotasowski/work/test/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/xs/xfp1z4cn643c46w69lv8bv8w0000gp/T/go-build4163821748=/tmp/go-build -gno-record-gcc-switches -fno-common'

Scenario 1 - creating a new module and downgrading go

What did you do

$ mkdir test && cd test
$ go mod init blah
$ go get go@1.21

What did you see happen?

• Tool chain directive of 1.22.0 was added - but this isn't necessary
• min go version set to 1.21.7

What did you expect to happen?

• No toolchain directive to be added
• Min go version set to 1.21 - since I didn't specify the point release

Scenario 2 - pulling a new dependency that upgraded the go version

What did you do

Project structure

--- go.mod
module blah

go 1.19

---- main.go
package main

import "fmt"

func main() {
	fmt.Println("vim-go")
}

Fetch a dependency that updates the go min version

$ go get knative.dev/pkg@9f033a7

What did you see happen?

console output

go: upgraded go 1.19 => 1.21
go: added toolchain go1.22.0
go: added knative.dev/pkg v0.0.0-20240221065059-9f033a7b77f7

go.mod output

module blah

go 1.21

toolchain go1.22.0

require knative.dev/pkg v0.0.0-20240221065059-9f033a7b77f7 // indirect

What did you expect to happen?

toolchain directive shouldn't appear. We want it to always default to the same as the go directive

[bcmills]

Please fill out the complete issue template.

In particular: what version of Go are you using, what was your go directive set to initially, and what commands did you run that added the toolchain directive?

[dprotaso]

@bcmills updated - can you take a look again and see if I'm missing anything?

[dylan-bourque]

Adding my own experiences for reference.

At work, we would definitely prefer to have the ability to disable the automatic toolchain upgrades (and possibly fail builds that would otherwise require a newer toolchain). We have many modules with the go directive in their go.mod files targeting Go versions from 1.16 to 1.21. We also do not proactively update those directives across the codebase when a new Go version is released.

This feature started causing issues for us when one engineer using Go 1.21.x locally updated some dependency and ran go mod tidy, which silently updated the go directive in that module and added the new toolchain one. We were all surprised when this change failed to build in CI, which was still using Go 1.20.x.

Without the ability to opt out of automatic toolchain upgrades we are now forced to synchronize the Go version across CI and every developer machine. While that's not an unreasonable stance in the general case, it is a new restriction/requirement that, as far as I've seen, is not called out in the release notes.

We have tried setting GOTOOLCHAIN to local but that doesn't seem to affect the behavior of go mod tidy. It will still upgrade both the go and toolchain directives.

One could also argue that forcing more conservative development shops to always stay on the most current Go release, despite the previous minor release being explicitly supported, is a bad situation.

[bcmills]

As of Go 1.21, the initial toolchain release has a .0 suffix (see #57631), so go 1.21 is a development version of the language — not a released version.

And for the toolchain directive in particular, per https://go.dev/doc/toolchain: “For repeatability, any command that updates the go line also updates the toolchain line to record its own toolchain name.”

Some specifics:

• go get knative.dev/pkg@9f033a7 is adding the toolchain line because it is upgrading the go line. That seems to be working as designed.

• go get go@1.21 is resolving to a specific point release because that's what it usually does for version prefixes. Unfortunately, the language version is a prefix of the release version, and we don't have a good way to force the language version instead of resolving the prefix. That probably needs some more thought.

• I don't understand why go get go@1.21 is not writing out a toolchain line. That is at the very least a mismatch between the documentation and the implementation.

[bcmills]

At work, we would definitely prefer to have the ability to disable the automatic toolchain upgrades (and possibly fail builds that would otherwise require a newer toolchain).

You do have that ability: you can set GOTOOLCHAIN=local in your process environment or GOENV file. (You can even edit $GOROOT/go.env to make that the default for your Go toolchain installation, if you are so inclined.)

You can also force go get to stick to a particular go version, the same way you would for any other dependency that you want to avoid upgrading, by passing that version as an explicit argument to go get: go get go@1.20 will downgrade to go 1.20, and go get knative.dev/pkg@9f033a7 go@1.20 will correctly error out (and report that those versions are not mutually compatible).

Without the ability to opt out of automatic toolchain upgrades we are now forced to synchronize the Go version across CI and every developer machine.

That is exactly the problem that GOTOOLCHAIN=auto is supposed to mitigate: when everyone is on Go 1.21.0 or above, the developers' machines should automatically download whatever toolchain is needed to work in the module.

[dylan-bourque]

You do have that ability: you can set GOTOOLCHAIN=local in your process environment

This works, but it also means that as soon as anyone moves to a newer Go release then everyone else in the organization must also move to that same newer version immediately. This is a problem in tightly regulated environments where, as an example Go 1.22.x may not be "certified" but Go 1.21.5 is. As it stands today it's not possible to both fix CI to 1.21.x and also allow developers to use a newer 1.21.y or 1.22.

You can also force go get to stick to a particular go version ... by passing that version as an explicit argument to go get: go get go@1.20

TIL. Thanks for this.

[cevich]

you can set GOTOOLCHAIN=local in your process environment

There are environments where setting an env. var. is not possible. For example, the public Renovate service does not support this. There are other ways to pin the golang version in Renovate, but they're undesirable for maintainability reasons. Devcontainers could be another example, where the tooling env. is shared, and must remain static, for consistency across a whole team of developers. In any case, my point is there are places where setting env. vars. isn't a viable workaround. IMHO such major behavior changes should almost always default to the previous or "least disruptive" option.

[gaby]

I just ran into this same issue, go1.21 is supported but my box has go1.22. I can't run any commands because golang keeps forcing the toolchain directive into the go.mod file.

This is for a library used by a lot of users and it should support go1.21 and go1.22

[XANi]

Still a problem, it just makes a complete mess any time common CI server is used by few people

[dmitshur]

I'm not seeing this already mentioned here, but in situations where you're looking to change the go directive and also not have a toolchain directive, have you considered explicitly including toolchain@none in the go get command? For example:

$ cat go.mod
module example.com

go 1.21.0

$ go get go@1.22.6 toolchain@none
go: upgraded go 1.21.0 => 1.22.6

That comes with the disadvantages of not having the toolchain version written down, but that's no different from the outcome where it's manually removed after go get adds it.

[prymitive]

As of Go 1.21, the initial toolchain release has a .0 suffix (see #57631), so go 1.21 is a development version of the language — not a released version.

A while ago one would set the go field in go.mod to specify the language version the project requires at minimum. The change to use full go release as the value of go, rather than just the major/minor release numbers as before, means that every single module might specify a different point release when in reality all patch releases of Go X.Y are guaranteed to be compatible - and if they are not that would be for some very good reason and is rare.
So every library out there putting a different point release in their go.mod combined with this automatic bumping of go / toolchain means there's a lot of noise in PR diffs that does not (in my opinion) adds any value at all. It's just churn.

And for the toolchain directive in particular, per https://go.dev/doc/toolchain: “For repeatability, any command that updates the go line also updates the toolchain line to record its own toolchain name.”

Some specifics:

• go get knative.dev/pkg@9f033a7 is adding the toolchain line because it is upgrading the go line. That seems to be working as designed.
• go get go@1.21 is resolving to a specific point release because that's what it usually does for version prefixes. Unfortunately, the language version is a prefix of the release version, and we don't have a good way to force the language version instead of resolving the prefix. That probably needs some more thought.

A while ago go.mod would be the files that's maintained / validated by the code author to write down requirements and constrains of the project. The go.sum file would be the lock file that holds all the low level details of how to resolve this constrains / requirements.
With this automatic bumping of go and insertion/updates of toolchain go.mod starts to also be a mini-lock file that encodes some of that information, which I think is unfortunate.
The decision for all of that happens automatically, unless the use puts in place enough env variables or files that stops go command from doing all of this means that there's by default a magic happening where it wasn't before, and maybe all of that will get easier and more obvious over time, but at present (for me personally) it took away simplicity of go.mod away while not giving me anything in return.

[dnwe]

I'm not seeing this already mentioned here, but in situations where you're looking to change the go directive and also not have a toolchain directive, have you considered explicitly including toolchain@none in the go get command?

Thanks for posting this. That's exactly the behaviour I (and probably others) had been looking for and was originally the behaviour that I thought that GOTOOLCHAIN=local would provide for me — i.e., always use the toolchain I have installed locally and never add a toolchain directive in the go.mod file. I always have the most recent release kept up-to-date locally, but I don't want that to keep getting stamped into the go.mod when its unnecessary.

Certainly for my own individual modules I'd likely move to a model of always doing:
go get toolchain@none go@{{- .GoLangVersion }}.0 <otherpkg> [otherpkgs...] to retain the behaviour of the go directive declaring the minimum language and stdlib version required to build the module, and then I'm happy for whatever toolchain >= to that version to be used at build time.

Ideally I'd want to bake that client behaviour into the go.mod file so that when anyone else updates a dependency, it retains the go directive and doesn't add an arbitrary toolchain directive based on whatever version they happened to have installed locally.

[matthewhartstonge]

I would argue that by adding toolchain as a non opt-in feature breaks user expectations around Go's release policy due to shadow forcing people on to whatever latest version is injected into go.mod.

Release Policy¶
Each major Go release is supported until there are two newer major releases. For example, Go 1.5 was supported until the Go 1.7 release, and Go 1.6 was supported until the Go 1.8 release. We fix critical problems, including critical security problems, in supported releases as needed by issuing minor revisions (for example, Go 1.6.1, Go 1.6.2, and so on).

I have found this really painful for library developers as well. I've had to ask maintainers to lower the versioning or remove the toolchain directive in a popular OSS lib to support said release policy.

As with others, I'm finding I'm constantly wasting time having to unwind/remove the toolchain directive as we run a n-1 Go versioning policy at our place of work.

[XANi]

It's so bad I'm considering adding git filter for it. Every time anyone runs even minor release difference compared to build server it breaks

[emmaLP]

Any update on this by any chance?

Tried upgrading one of my modules to 1.23 (previously 1.22 and not had a problem before), but every go mod tidy is failing because it's setting the toolchain and without it our CI breaks.

We deliberately don't set the toolchain so that we can get patch versions of a specific golang version.

I've tried setting on my local machine the GOTOOLCHAIN to be local as well as tried auto and it always sets the go directive to 1.23.0 and the toolchain 1.23.1.

Only way I've managed to stop the toolchain directive been added to to set my go directive to explicitly be go 1.23.1. This defeats the point of auto upgrading to the latest patch version.

[andrewmed-prop]

Frankly, over the many years I have not seen anything so intrusive, from the go authors:

-It (the locally installed compiler) goes to internet itself (!)
-It updates the project files with unnecessary information (info on my toolchain)
-It can not be disabled

Even if backward compatibility is no more a guarantee for go (which I think is not the case) this feature should have an option to be disabled. In out setup we are just fine with setting up the minimum version, as it always has been the case.

ps. I do not want really think this way, but can not help imagining it is something telemetry-related...

[jdemeyer]

And for the toolchain directive in particular, per https://go.dev/doc/toolchain: “For repeatability, any command that updates the go line also updates the toolchain line to record its own toolchain name.”

Maybe it would help to explain better why the toolchain line is being added: what kind of "repeatability" does it ensure?

[XANi]

so far it mostly ensures repetably breaking any kind of development setup with more than one machine...

edit: honestly using go dependency management just becomes MORE frustrating over time once "features" are added.

forget to push lib's tag before trying to pull it in another project ? enjoy reading https://sum.golang.org/lookup/github.com/something@v1.4.8: 404 Not Found, because apparently for some fucking reason it needs to check up with mothership rather than actually try to download actually available tag that was just pushed to repo 5 seconds later

dare to develop something without repo open on the internet ? Enjoy having to set up GOPRIVATE on every fucking device that will need to compile it ever instead of any kind of reasonable graceful fallback of just downloading a fucking file like it did before those "features" were added.

Want to have one version on build server and not update it any time any of your devs dares to update system ? Tough luck, maybe set up a hook to kick off the toolchain directive

/rant over. And still too short for sum.golang.org negative caching to fuck off...