all: repeatedly running all.bash in an unchanged tree grows the Go build cache without limit

[siebenmann]

What version of Go are you using (go version)?

$ go version
go version devel go1.22-890b96f7ab Thu Jul 20 08:03:42 2023 +0000 linux/amd64

Does this issue reproduce with the latest release?

This issue doesn't reproduce when building Go 1.20.6 from a 'git checkout go1.20.6' in the source tree.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/cks/.cache/go-build'
GOENV='/home/cks/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/cks/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/cks/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/w/620/cks/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/w/620/cks/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='devel go1.22-890b96f7ab Thu Jul 20 08:03:42 2023 +0000'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/w/620/cks/go/src/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3572262321=/tmp/go-build -gno-record-gcc-switches'

What did you do?

git clone https://go.googlesource.com/go
cd go/src
./all.bash
du -hs ~/.cache/go-build
./all.bash
du -hs ~/.cache/go-build
[repeat]

The first all.bash from source will create about 2.3 GB of Go build cache. Subsequent all.bash runs appear to increase this by about 200 Mbytes each time. If you repeatedly build for long enough (if, for example, you're using this as a little stress test on a new system), this will fill up your $HOME with hundreds of GBytes of ~/.cache/go-build. 'go clean -cache' cleans it all away.

[heschi]

cc @bcmills @matloob

[ianlancetaylor]

Just a note that every 24 hours the cache is checked for old entries, and entries that are older than 5 days are deleted. So in the long run there is a limit.

But it does seem odd that running all.bash with the same sources would increase the cache size.

[bcmills]

@siebenmann, some questions:

• Does the cache grow when you run make.bash, or only all.bash?
• Does the output of go tool compile -V=full change between runs?
• Do you see similar growth if you run go test -short std cmd?

[bcmills]

Specifically, I wonder if that ~200 MB of data per run is due to caching packages built from within test temp directories. If so, maybe some tests should be updated to build with -trimpath or similar to get the temp directories out of the cache keys.

[bcmills]

(There may be a cmd/go bug here as well — for example, it is possible that this data is in package indexes, which IIRC currently contain absolute paths.)

[siebenmann]

This growth doesn't happen with just 'make.bash' (which only seems to use 455 MBytes or so from a clean cache). Running go test -short std cmd repeatedly doesn't seem to cause growth, but I'm not sure I'm doing it right because it fails (when all.bash succeeds). There's no difference between go tool compile -V=full outputs across multiple 'all.bash' runs, despite the build cache growing.

In terms of the cache's growth rate versus periodic trimming, the cache size can reach hundreds of GBytes (I saw over 350 GB) if you're running 'all.bash' in a loop repeatedly.

[bcmills]

Running go test -short std cmd repeatedly doesn't seem to cause growth, but I'm not sure I'm doing it right because it fails (when all.bash succeeds).

Huh. Could you file a separate issue for the failures you observe from that? (It ought to work.)