Add support for Go 1.21 toolchains

[eskultety]

Go 1.21 toolchains

Go 1.21 introduced a new feature called toolchains which allow usage of different Go toolchains to be used for different modules simultaneously instead of the default bundled Go toolchain (i.e. the bundled go binary). The toolchain feature is enabled by the toolchain keyword which can either be in the go.mod file or the go.work (workspaces) file [1].

Important things to keep in mind

• the go line in go.mod denotes the minimum required Go version to use to compile a module
• the toolchain line denotes the suggested go toolchain version to be used
  • however, the suggested toolchain version must not be older than the minimum required Go version [2]
• different modules and workspaces can denote different versions of toolchains
• Go always ignores the toolchain keyword if that version is less than the bundled Go's version

Controlling toolchain behaviour

When it comes to toolchains Go's behaviour depends primarily on the GOTOOLCHAIN environment variable which can be set in a few different places. Go's lookup for environment variables goes as follows:

1. the running process' environment is checked for variables (e.g. GOTOOLCHAIN=<selector> go )
2. $GOENV variable is checked for user's environment configuration file [3] (e.g. it may point to /home/user/.config/go/env)
3. $GOROOT/go.env is checked; this is the global installation environment file and its location is platform-dependent
   • distro repackaged Go releases may freely set variables differently from the official vendored releases [4]
4. GOTOOLCHAIN=local is used as the default

GOTOOLCHAIN selector values

The selector is what determines Go's behaviour when it comes to respecting the toolchain setting in go.mod or go.work files.
The value GOTOOLCHAIN accepts could be roughly described as GOTOOLCHAIN=<selector>(+<selector>)? where
<selector> can be one of:

• local - always run the default bundled toolchain; it's often paired with another selector, e.g. local+path which means that whenever possible Go should prefer the alternative toolchain
• name - always runs toolchain with that specific name; Go will look into its PATH and if such toolchain binary is missing it'll download it from the internet)
• path - same as name but DOES NOT download the toolchain if Go couldn't find it in PATH; in its bare form it's a shortcut for local+path
• auto - shortcut for local+auto, i.e download new toolchains as needed

For our purposes though we should only ever be concerned with GOTOOLCHAIN=auto and pre-fetch whatever toolchain version Go encounters during processing.

Downloading toolchains

Luckily toolchains are nothing else than a gomod dependency which is cached like any other Go dependency. Unfortunately for us:

1. Go by default never looks into the mod cache for toolchains (not entirely true, read on...) and so it always downloads it anew
2. Go always wants to perform (not entirely true, read on...) checksum validation on the downloaded toolchains
   • checksum validation is controlled by the GOSUMDB variable (see Other Affecting Variables)

Go always performs checksum validation on toolchains

Starting with the checksums first as the downloads may be more clear later. So GOSUMDB was mentioned to control the checksum validation, so if GOSUMDB=off (Fedora default) what happens with either go mod download or go build while suggesting a toolchain to be used is that Go goes and fetches the toolchain from the internet, but then fails the checksum verification and so it refuses to proceed further with anything (e.g. downloading other dependencies or building anything) (see Fetching toolchain with checksum disabled). Therefore, to support Go 1.21 toolchains GOSUMDB needs to be always set explicitly by us and point to a valid checksum server as long as GOTOOLCHAIN is set to anything but local which forces Go to always use the bundled toolchain (see GOTOOLCHAIN selectors).

Note there's also the GONOSUMDB variable (see Other Affecting Variables) which might give the impression that if we specify a pattern for toolchains, we should be able to either download toolchains insecurely (not that we want to!). Turns out toolchains are exempt from any checksum verification settings [5] and so we need to account for this.

Go always downloads the toolchain

With setting GOSUMDB explicitly we only takes care of a part of the puzzle, because it only ensures that we'll pre-fetch a toolchain correctly, but will not make sure in any way that such toolchain is actually going to be used during the project's container build or even that such a container build would succeed in general, because Go will ultimately try to download it again (except it won't, because builds may be hermetic).
To solve the fetch/cache/use issue, one needs to can set the GOPROXY variable as well so that it points to the cache on the local file system [6]. That way Go will continue using its original logic and will fetch the toolchain from the pre-fetched dependencies cache (into the same cache btw, fill in your favourite meme...) instead of pulling it from the internet. So far so good, back to checksums. So, we already know that Go does checksum verification on downloaded toolchains by pointing it to a valid checksum server, so how do we get them verified during hermetic builds? Turns out, Go only performs toolchain checksum verification when pulling from the internet, but not when using the file:// URL scheme inside GOPROXY \o/. In other words, Go only tries to verify remote resources, but somehow trusts local resources (see Fetching with GOMODCACHE and GOPROXY).

And now the "Crème de la crème" of Go toolchain downloads and verification. Remember the section intro about Go seemingly never looking into GOMODCACHE for toolchains? It actually does, but guess what, it needs to verify the toolchains checksum (see Use toolchain with an offline cache.

Example invocations

This section serves just as a reference to various invocations of Go forcing the toolchain usage.

Fetching toolchain with checksum disabled

$ GOTOOLCHAIN=auto go mod download
go: downloading go1.21.2 (linux/amd64)
go: download go1.21.2: golang.org/toolchain@v0.0.1-go1.21.2.linux-amd64: verifying module: checksum database disabled by GOSUMDB=off

# try again to showcase Go will download the toolchain again (but let it pass the verification)
$ GOTOOLCHAIN=auto GOSUMDB=sum.golang.org go mod download
go: downloading go1.21.2 (linux/amd64)
...
go: downloading golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c
# get https://proxy.golang.org/golang.org/x/text/@v/v0.0.0-20170915032832-14c0d48ead0c.zip
# get https://proxy.golang.org/golang.org/x/text/@v/v0.0.0-20170915032832-14c0d48ead0c.zip: 200 OK (0.007s)
# get https://proxy.golang.org/golang.org/x/text/@v/v0.0.0-20170915032832-14c0d48ead0c.mod
# get https://proxy.golang.org/golang.org/x/text/@v/v0.0.0-20170915032832-14c0d48ead0c.mod: 200 OK (0.005s)
...

Use toolchain with an offline cache

$ GOTOOLCHAIN=auto GOSUMDB=off GOMODCACHE=/tmp/gomod/ go mod download --json
go: golang.org/toolchain@v0.0.1-go1.21.2.linux-amd64: verifying module: checksum database disabled by GOSUMDB=off

Fetching with GOMODCACHE and GOPROXY

$ GOTOOLCHAIN=auto GOSUMDB=off GOMODCACHE=/tmp/gomod/ GOPROXY=file:///tmp/gomod/cache/download go mod download --json
    {
	"Path": "golang.org/x/text",
	"Version": "v0.0.0-20170915032832-14c0d48ead0c",
	"Info": "/tmp/gomod/cache/download/golang.org/x/text/@v/v0.0.0-20170915032832-14c0d48ead0c.info",
	"GoMod": "/tmp/gomod/cache/download/golang.org/x/text/@v/v0.0.0-20170915032832-14c0d48ead0c.mod",
	"Zip": "/tmp/gomod/cache/download/golang.org/x/text/@v/v0.0.0-20170915032832-14c0d48ead0c.zip",
	"Dir": "/tmp/gomod/golang.org/x/text@v0.0.0-20170915032832-14c0d48ead0c",
	"Sum": "h1:qgOY6WgZOaTkIIMiVjBQcw93ERBE4m30iBm00nkL0i8=",
	"GoModSum": "h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ="
}
...

Proposed solutions

1. Ignore toolchains alltogether

Since we only need to control Go's behaviour using environment variables, these are the necessary settings:

Dependency pre-fetch Go configuration

Set the following on top of our existing env settings:
GOTOOLCHAIN=local

User container build Go configuration

Set the following on top of our existing env settings:
GOTOOLCHAIN=local

Advantages:

• pretty much no real work needed on our side

Disadvantages:

• may not shield us properly from users wanting the toolchain feature in future, i.e. someone may come complaining (they will!)
• we'll have to keep up with frequent Go releases and update the container image promptly, in other words continue what we do now

2. Pre-fetch ALL affecting toolchains

Let Go download all toolchains it may need (for all modules). Since we only need to control Go's behaviour using environment variables, these are the necessary settings:

Dependency pre-fetch Go configuration

Set the following on top of our existing env settings:
GOTOOLCHAIN=auto
GOSUMDB=sum.golang.org (we already set this one...)

User container build Go configuration

Set the following on top of our existing env settings:
GOTOOLCHAIN=auto
GOSUMDB=off
GOPROXY=file://${GOMODCACHE}/cache/download

Advantages:

• we'll be properly supporting Go 1.21 features going in the future
• in the very unprobable theory we might not need to keep bumping Go versions in our container image that often if users make use of toolchains properly, because if they set go 1.21 and then toolchain 1.24.0 (provided 1.24 is out) then the 1.21 binary will be able to download a newer toolchain and use it for compiling; however, this whole idea crumbles like a house of cards when it comes to features that include new keywords

Disadvantages:

• potentially failing builds due to transitively incompatible (i.e. with indirect dependencies) go and toolchain versions set - but this is a user problem in general though, not cachi2's (mentioning it just in case)

References

[1] standard Go rules apply when it comes to evaluating modules vs workspaces
[2] there are some strict rules when it comes to versioning of suggested toolchains which applies transitively too across all dependencies, but luckily that's not our problem to figure out! As noted in https://go.dev/doc/toolchain:

A module’s go line must declare a version greater than or equal to the go version declared by each of the modules listed in require statements. A workspace’s go line must declare a version greater than or equal to the go version declared by each of the modules listed in use statements.

[3] the go.env file has a simple KEY=VALUE structure
[4] Fedora repackaged go.env:

$ cat /usr/lib/golang/go.env

# This file contains the initial defaults for go command configuration.
# Values set by 'go env -w' and written to the user's go/env file override these.
# The environment overrides everything else.

# Use the Go module mirror and checksum database by default.
# See https://proxy.golang.org for details.
GOPROXY=direct
GOSUMDB=off

# Automatically download newer toolchains as directed by go.mod files.
# See https://go.dev/doc/toolchain for details.
GOTOOLCHAIN=local

Go's default vendored go.env :

# This file contains the initial defaults for go command configuration.
# Values set by 'go env -w' and written to the user's go/env file override these.
# The environment overrides everything else.

# Use the Go module mirror and checksum database by default.
# See https://proxy.golang.org for details.
GOPROXY=https://proxy.golang.org,direct
GOSUMDB=sum.golang.org

# Automatically download newer toolchains as directed by go.mod files.
# See https://go.dev/doc/toolchain for details.
GOTOOLCHAIN=auto

[5] Snippet from the docs:

toolchain downloads fail for lack of verification if GOSUMDB=off. GOPRIVATE and GONOSUMDB patterns do not apply to the toolchain downloads.

[6] Snippet from the docs:

A module cache may be used directly as a file proxy: GOPROXY=file://$(go env GOMODCACHE)/cache/download

Resources

https://go.dev/doc/toolchain
https://go.dev/ref/mod#environment-variables

Apendix

Other affecting variables

There are a couple of other variables that further affect module checksum verification and the download process:

• GOSUMDB - Identifies the name of the checksum database to use and optionally its public key and URL, normally defaults to sum.golang.org, on Fedora it's "off" [5]
• GONOSUMDB - Comma-separated list of glob patterns of module path prefixes for which the go should not verify checksums using the checksum database
• GOPROXY - a (comma separated) list of URLs pointing to module proxies (instead of downloading modules directly from VCS); it supports the following URL schemes:
  • https/http
  • file
• GOPRIVATE - list of glob patterns of module path prefixes that should be considered private. Acts as a default value for GONOPROXY and GONOSUMDB. This is useful when no proxy serves private modules

[eskultety]

@brunoapimentel @taylormadore @ben-alkov @ejegrova @slimreaper35 please have a look and leave comments.

[ben-alkov]

Approvals

• @ben-alkov
• @brunoapimentel
• @ejegrova
• @slimreaper35
• @taylormadore

[brunoapimentel]

1. Ignore toolchains alltogether
   explicitly set GOTOOLCHAIN=local and rely solely on the minimum required Go version to build the project

Just to clarify my understanding: what happens if the repo has a module that declares toolchain go1.22 and it was not prefetched? I assume the build would fail because go will try to fetch it from the Internet. So would we set GOTOOLCHAIN=local in the cachi2.env file to force the build to only use the currently available local go binary?

Assuming I got the above right, I believe there could be some corner cases in which the build would fail because some module can't be built with a different version of go than what's in the toolchain (e.g. some new feature was added to a newer go version, and this specific module requires it). My only concern here is that this build error could be hard to troubleshoot, it would be nice to have some guidance to the user.

[brunoapimentel]

I'm wondering if we could use solution 1 as a way to release 1.21 quickly, and keep the proper toolchain support (solution 2) in our backlog.

[eskultety]

Just to clarify my understanding: what happens if the repo has a module that declares toolchain go1.22 and it was not prefetched? I assume the build would fail because go will try to fetch it from the Internet. So would we set GOTOOLCHAIN=local in the cachi2.env file to force the build to only use the currently available local go binary?

Well, if we don't do anything, then it solely depends on the platform's GOTOOLCHAIN setting. Assuming we'd build on Fedora where GOTOOLCHAIN=local the bundled go toolchain would always be used provided we meet the minimum required Go version inside the builder container

Assuming I got the above right, I believe there could be some corner cases in which the build would fail because some module can't be built with a different version of go than what's in the toolchain (e.g. some new feature was added to a newer go version, and this specific module requires it).

That's something we have no control over and is clearly a user error because that's what the go line is for, so they have to make sure they set the minimum required version of Go to build their module correctly. One simply cannot rely on the fact that the toolchain setting will be honored unless they make sure the variable is set explicitly which cannot even be done inside the repo so that Go would read the env file by default, it has to be pointed to explicitly. However, to make this reproducible and deterministic, we'd have to set the GOTOOLCHAIN variable explicitly ourselves in cachi2.env as you say.

If it wasn't clear from the original description, I'm attaching another argument-supporting snippet from the docs:

===================================

That is, the go line sets the minimum required Go version necessary to use a module or workspace.
...
The toolchain line declares a suggested toolchain to use with the module or workspace. As described in “Go toolchain selection” below, the go command may run this specific toolchain when operating in that module or workspace if the default toolchain’s version is less than the suggested toolchain’s version.

===================================

My take on the whole toolchain thing is that because it's a suggested version, one can start preparing their project to moving to the next minor Go release, i.e. set the minimum required version to 1.21, but try to safely build with 1.22 e.g. in a CI environment to see if some new compiler feature breaks the build and changes are going to be required to the project. This might sound weird, but that's what we did in a different project I worked on with GCC - we used a newer GCC (typically Rawhide) in CI to see if we can build safely with the coming versions of GCC and fix problems early before they appear in the mainstream release. I might be wrong but the way toolchains are defined in Go it sounds like one would want to do a very similar thing.

I'm wondering if we could use solution 1 as a way to release 1.21 quickly, and keep the proper toolchain support (solution) in our backlog.

You're reading my mind here @brunoapimentel :).

[brunoapimentel]

Well, I'm late to the party, that snippet you pasted was in the blog article I was reading this morning 😮‍💨

This means that this advantage here is indeed very unlikely...

in the very unprobable theory we might not need to keep bumping Go versions in our container image that often if users make use of toolchains properly, because if they set go 1.21 and then toolchain 1.24.0 (provided 1.24 is out) then the 1.21 binary will be able to download a newer toolchain and use it for compiling;

...since what I've seen users do is simply bump the go version when a new one is available, which would now make the prefetch fail completely until we also update Cachi2.

however, this whole idea crumbles like a house of cards when it comes to features that include new keywords

When there are breaking changes between go versions, bumping the minimum go version is the correct approach, so failing to do so would classify as a misconfigured repository :)

[brunoapimentel]

Well, I guess it makes sense to start with solution 1, than move to solution 2 when we have more time.

[eskultety]

Well, I guess it makes sense to start with solution 1, than move to solution 2 when we have more time.

When I get the ACKs, I can post it right away. Note that even the latter wouldn't be too much of a hassle :).

[ben-alkov]

Checkboxes for approvals are up there ⬆️

[brunoapimentel]

I think it would make it clearer if we specified in the solutions which environment variables are meant for build time, and which ones are meant for the prefetch.

[eskultety]

Reworked - how about now?

[brunoapimentel]

Looks great!

[eskultety]

There was an issue that popped up recently about Golang 1.21. With Golang 1.21, the go.mod file is updated if the new 1.21 toolchain detected that the main module's minimum required Go version (denoted by the go X.Y line) is less than the one declared by ANY of its dependencies. This finding is supported by the following snippet in the doc reference (thank you @taylormadore for digging it out):

At go 1.21 or higher:

The go line declares a required minimum version of Go to use with this module.
The go line must be greater than or equal to the go line of all dependencies.
The go command no longer attempts to maintain compatibility with the previous older version of Go.
The go command is more careful about keeping checksums of go.mod files in the go.sum file.

This complicates matters for cachi2 significantly, because what happens with projects which specify their minimum required Go version as let's say 1.20, but any of it dependencies declare 1.21, go would update the go.mod file of the project which is kind of an undesired side-effect (although it's arguable that it could be a user error in the sense that they blindly updated their dependencies without checking what golang version those require not taking into consideration the consequences with more modern toolchain versions).

However, here's a proposal that should in theory take care of the problem (although I'm not proud of it in any way, because it'd add maintenance burden, although how big?):

1. bump the container version of Golang to default to 1.21
2. parse the project's go.mod file by using go mod edit -json and extract the minimum/suggested (prior to 1.21) version

{
	"Module": {
		"Path": "golang.org/dl"
	},
	"Go": "1.18",
	...
}

3. if the version is <1.21 enter an alternative code path which:
   3.1. downloads a fixed toolchain version, say 1.20.0 with go install golang.org/dl/go1.20.0@latest
   3.2. fallback to using an older version of Go to prevent any unwanted side-effect repository changes made by Go as a result of running cachi2

4. continue as normal

[chmeliik]

This is pretty much what I was going to suggest. If go 1.21 breaks the backward compatiblity that cachi2 relied on, use go 1.20.

Maybe download go 1.20 while building the image, not at runtime? Downloading the latest go 1.20 at runtime feels scary (and would take some time).

FWIW go 1.20 should be EOL soon-ish (https://go.dev/doc/devel/release#policy), so if the incompatibility is only between <1.21 and 1.21, the need for this might go away eventually? Unless 1.22 is incompatible with 1.21 as well 😨

[ben-alkov]

Unless 1.22 is incompatible with 1.21 as well

"The go command no longer attempts to maintain compatibility with the previous older version of Go" might as well be Go's official slogan...

[eskultety]

Maybe download go 1.20 while building the image, not at runtime? Downloading the latest go 1.20 at runtime feels scary (and would take some time).

If we do that then purely CLI use case would still suffer from this. The download only took a couple of seconds, so not THAT bad (for a temporary solution...sorry, too soon?) Why do you think it's scary? It's the same as with downloading any other module from the official proxy and it's also checksum verified since it's done by Go directly.

[chmeliik]

If we do that then purely CLI use case would still suffer from this.

Ah, that's true.

The scary part to me is that cachi2 will download something not entirely verified and then execute it. But yeah, downloading go through the official proxy and relying on the official gosumdb for checksum verification should make it safe enough. It's not any worse than what cachi2 does for yarn, anyway.

[eskultety]

It's not any worse than what cachi2 does for yarn, anyway.

and is also no different from pulling Go mod dependencies through the same proxy within the same step. Also note that with toolchains and with GOTOOLCHAIN=auto the only major difference there to downloading a different, say older, version of Go is automation - to download a Go toolchain prior 1.21 we have to do it manually, after 1.21 Go does it on its own.

[eskultety]

@brunoapimentel @ejegrova @taylormadore @slimreaper35 are we all in agreement with the approach proposed in this discussion, including the necessary fix proposed in #419 (comment)?

If so, please consider an ACK: #419 (reply in thread)