cmd/compile: Go 1.9 oom killed on compilation #21713
Labels
FrozenDueToAge
NeedsDecision
Feedback is required from experts, contributors, and/or the community before a change can be made.
Milestone
Comments
|
How much ram does your system have? Can you please include the relevant oom
message from dmesg(1)
…On Thu, 31 Aug 2017, 17:13 Yannick Briffa ***@***.***> wrote:
What version of Go are you using (go version)?
1.9
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user"
GORACE=""
GOROOT="/home/ybriffa/go"
GOTOOLDIR="/home/user/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0
-fdebug-prefix-map=/tmp/go-build501174937=/tmp/go-build
-gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
What did you do?
I compiled a very large code base (over 2000) of generated files,
essentially composed of structure definition.
What did you expect to see?
Not to crash.
What did you see instead?
All my RAM filled, then my swap and finally the compilation got killed.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#21713>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAAcA3SWJJRH2HXr6KbpZU6_JOsuP-M0ks5sdl0jgaJpZM4PIaeH>
.
|
|
I have 4Go RAM + 1Go swap An information I forgot: I had no troubles under 1.8 |
|
Try the -p 1 flag to limit the number of packages compiled in parallel.
There is an outstanding issue to limit the number of files parsed in
parallel which may also help once it lands.
…On Thu, 31 Aug 2017, 17:38 Yannick Briffa ***@***.***> wrote:
I have 4Go RAM + 1Go swap
An information I forgot: I had no troubles under 1.8
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#21713 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAcAwTIwU306PyhYc7U2CF4BCTPvejJks5sdmLpgaJpZM4PIaeH>
.
|
|
it does not change anything. At the begining the compilation goes well, and at what looks like to be the end (maybe at the linking), the RAM and swap are filled and I get oom killed |
|
If you can post the OOM message it will help
…On Thu, 31 Aug 2017, 17:56 Yannick Briffa ***@***.***> wrote:
it does not change anything. At the begining the compilation goes well,
and at what looks like to be the end (maybe at the linking), the RAM and
swap are filled and I get oom killed
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#21713 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAcAxzz7kJVmc2RkpftvN7SgackmwhLks5sdmcQgaJpZM4PIaeH>
.
|
|
Sure, here it is: |
|
The output from dmesg(1) please.
…On Thu, Aug 31, 2017 at 6:27 PM, Yannick Briffa ***@***.***> wrote:
go build stash.ovh.net/apiv7lib/modelsources:
/home/user/go/pkg/tool/linux_amd64/compile: signal: killed
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#21713 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAcAzkzHaqXM6ntp-Kmk3R_Lc-thgc7ks5sdm6HgaJpZM4PIaeH>
.
|
|
read kernel buffer failed: Operation not permitted I'm running under a dockerized environment |
|
Is it possible for you to build outside docker as a test? It's important to see how large the process gets before it is killed.
… On 31 Aug 2017, at 18:59, Yannick Briffa ***@***.***> wrote:
read kernel buffer failed: Operation not permitted
I'm running under a dockerized environement
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
I can try, by now, what I can tell (with htop data) is that my RAM usage before the compilation is 648/3941MB and my swap is 440/1187MB |
|
|
this one was under a 8Gb RAM + 1Gb Swap |
ianlancetaylor
changed the title
|
CC @mdempsky |
|
Did this code base compile with 1.8? Also, it would help if we could reproduce this issue ourselves. Are you able to share the code base with us, or to create a new demo repo that exhibits the same issue? |
|
This code base used to compile with 1.8. I'm going to look if I can create you a repo to reproduce this problem with not-sensitive data. I don' think it is related to the issue you've linked: I have exactly 2446 files. Also I tried to compile this code base over a continuous delivery software with a worker having 20Go RAM, it worked. However, I don't know exactly what is the threshold. |
|
After I have seen it with my company, I can provide you a tarball with the content of the code base a bit refined in order to debug. Since it is kind of sensitive content, where could I leave you the tarball ? |
|
@ybriffa Sorry for letting this slip. If it's not too large, you can just email it to me (mdempsky@google.com) as an attachment. Otherwise, upload to Google Drive or somewhere and share it with me. |
|
@mdempsky I've just sent you a tarball with the code we're using |
|
I received @ybriffa's test case. It's mostly self contained (only depends on github.com/mitchellh/mapstructure, which itself only depends on the standard library). It's 261015 lines of Go code spread over 2444 source files. Bisecting on @ybriffa's test case finds e22ba7f to be the cause of the slow down and memory growth. Before that commit, building the package takes ~30 seconds and 2GB of memory on my workstation. After that commit, it takes ~70 seconds and >8GB. According to -cpuprofile, it looks like most of the extra time is being spent in regalloc. |
|
I think the problem is there are two global |
|
This Go program generates a Go source file that demonstrates the issue: |
|
/cc @randall77 @josharian for e22ba7f |
|
Seems to be fixed at tip. Almost certainly CL 66050, the fix for #22010 which initializes large maps like this (constant string keys, particularly) with tables instead of code. Options:
I'm kinda leaning toward #3. These large files are certainly generated code (well, @mdempsky 's repro is, not 100% sure about the original bug). There is a workaround, just do what 1 does by hand, initialize an array instead and load the data into a map with a for loop. |
|
@randall77 Changing We should be able to table-initialize that too, since global funcs don't need closures. But it seems like we don't for some reason. |
|
@mdempsky I'll take a look at that. In any case, the question of what to do about 1.9 still stands. |
|
Change https://golang.org/cl/71210 mentions this issue: |
ALTree
added
the
NeedsDecision
|
@randall77, will let you decide but everyone at proposal-review is OK with option 3 "do nothing". |
|
I'll call this done then. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
What version of Go are you using (
go version)?1.9
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user"
GORACE=""
GOROOT="/home/user/go"
GOTOOLDIR="/home/user/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build501174937=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
What did you do?
I compiled a very large code base (over 2000) of generated files, essentially composed of structure definition.
What did you expect to see?
Not to crash.
What did you see instead?
All my RAM filled, then my swap and finally the compilation got killed.
The text was updated successfully, but these errors were encountered: