misc/cgo/testshared: segfaults on linux/arm64 Fedora 29

[jcajka]

What version of Go are you using (go version)?

go version devel +004fb5cb8d Fri May 3 03:49:11 2019 +0000 linux/arm64

Does this issue reproduce with the latest release?

Does reproduce on master, Go 1.12 and Go 1.11 on Fedora 29.

What operating system and processor architecture are you using (go env)?

linux/arm64

What did you do?

I have built master branch with go1.11.6 on linux/arm64 host(Fedora 29) and

go tool dist test -v -v -v -k

This is with

gcc-8.3.1-2.fc29.aarch64
binutils-2.31.1-17.fc29.aarch64
glibc-2.28-27.fc29.aarch64

What did you expect to see?

Tests passing

What did you see instead?

##### ../misc/cgo/testshared
# go tool dist test -run=^testshared$
go test -short -count=1 -tags= -run= -timeout=10m0s
--- FAIL: TestTrivialExecutable (8.15s)
    shared_test.go:41: executing ../../bin/trivial (trivial executable) failed signal: segmentation fault (core dumped):
--- FAIL: TestDivisionExecutable (3.22s)
    shared_test.go:41: executing ../../bin/division (division executable) failed signal: segmentation fault (core dumped):
--- FAIL: TestCgoExecutable (6.45s)
    shared_test.go:41: executing ../../bin/execgo (cgo executable) failed signal: segmentation fault (core dumped):
--- FAIL: TestGopathShlib (10.57s)
    shared_test.go:41: executing ../../bin/exe (executable linked to GOPATH library) failed signal: segmentation fault (core dumped):
--- FAIL: TestTwoGopathShlibs (11.90s)
    shared_test.go:41: executing ../../bin/exe2 (executable linked to GOPATH library) failed signal: segmentation fault (core dumped):
--- FAIL: TestThreeGopathShlibs (13.34s)
    shared_test.go:41: executing ../../bin/exe3 (executable linked to GOPATH library) failed signal: segmentation fault (core dumped):
--- FAIL: TestABIChecking (6.29s)
    shared_test.go:885: exe failed, but without line "abi mismatch detected between the executable and lib"; got output:
--- FAIL: TestImplicitInclusion (3.27s)
    shared_test.go:41: executing ../../bin/implicitcmd (running executable linked against library that contains same package as it) failed signal: segmentation fault (core dumped):
--- FAIL: TestInterface (4.50s)
    shared_test.go:41: executing ../../bin/iface (running type/itab uniqueness tester) failed signal: segmentation fault (core dumped):
--- FAIL: TestGlobal (3.58s)
    shared_test.go:41: executing ../../bin/global (global executable) failed signal: segmentation fault (core dumped):
shared_test.go:72: executing go test -installsuffix=6129484611666145821 -linkshared -test.short sync/atomic failed exit status 1:
exit status 1
FAIL	misc/cgo/testshared	111.018s
2019/05/03 10:43:35 Failed: exit status 1
run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}}	STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std
run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}}	STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE

[ianlancetaylor]

Odd. The builder isn't reporting any problems.

Can you reproduce the problem cd misc/cgo/testshared; go test -v? What is the output?

[jcajka]

Yeah, it looks odd. Possibly something wrong with the binaries/runtime. Build/test log looks sane to me, although failing. This unfortunately flew under my radar for some time :(.
Output of cd misc/cgo/testshared; go test -c -v; ./testshared.test -test.v
About to dig in to the binaries built during the test.

For the record what are the versions of binutils, glibc and gcc on builders? To rule out that angle.

[ianlancetaylor]

CC @bradfitz: see question about tools versions on builders. Thanks.

[ianlancetaylor]

Actually, never mind, I can use gomote to find the tool versions.

We're using gcc 4.9.2, binutils 2.25, glibc 2.19-18. So quite a bit older than the tools you are using. We are running Debian jessie.

[ianlancetaylor]

Thanks for sending the log output. Unfortunately I agree that it all looks normal. My first guess would of course be that the newer tools are causing a problem somehow, but I don't know why that would be. I think we'll need some way to reproduce the problem ourselves.

[bradfitz]

I can add a Fedora builder, but in the meantime (still commuting) I just updated our debian-sid builder. It hadn't been updated in quite some time, and it gcc was pretty old.

From 7.3.0 to 8.3.0:

bradfitz@go:~/src/golang.org/x/build/env/linux-x86-sid$ docker run -ti gcr.io/symbolic-datum-552/linux-x86-sid:latest /bin/bash
root@759b9aca4e34:/# gcc --version
gcc (Debian 7.3.0-18) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

bradfitz@go:~/src/golang.org/x/build/env/linux-x86-sid$ docker run -ti golang/linux-x86-sid /bin/bash
root@a6236231b605:/# gcc --version
gcc (Debian 8.3.0-7) 8.3.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Just deployed now.

[gopherbot]

Change https://golang.org/cl/175737 mentions this issue: env/linux-x86-sid: update our sid builder

[jcajka]

@ianlancetaylor Just did quick poke on it with gdb and the crash seems to happen during executable startup prior main function is invoked.

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x0000ffffbf2c8d24 in __libc_start_main () from /lib64/libc.so.6
#2  0x0000000000400a34 in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Intriguing is that the shared pie tests pass just fine.

To note I will continue digging in to this next week as I will be mostly offline for next 7 days.

[ianlancetaylor]

@bradfitz Note that this bug is for arm64. I can't tell whether you updated the arm64 builder or not.

[bradfitz]

Note that this bug is for arm64. I can't tell whether you updated the arm64 builder or not.

I just did linux-amd64-sid out of hope that updated tools would let us reproduce it alone. And we try to update the sid builder regularly anyway, so doesn't hurt.

But sadly that didn't catch it. We do have enough arm64 capacity that we could run a Fedora arm64 Docker container on it, assuming that'd be enough and it's not a kernel issue. That's a bit more tedious so I likely won't get to it right away.

[bradfitz]

I also tried Fedora 30 (also amd64) on a whim but it passed fine. I didn't try Fedora 29 on amd64, though.

[jcajka]

More digging in, this is actually dupe of #28334 and it has been/is binutils/gold issue https://sourceware.org/bugzilla/show_bug.cgi?id=23870 fixed on the trunk.
So closing. Thanks to nickc for quick backport of the patch for all Fedora releases.

[shawndx]

Hi,

Got the same issue on arm64, seems that it's caused (one of the reasons I think) by 'default pie' not being enabled, the issue could be reproduced by clang (6.0.0-1ubuntu2) and gcc without "--enable-default-pie" in my environment.

I have a preliminary fix which introduces '-extldflags=-pie' in gc-toolchain's ld part under a few conditions (arm64, building linkshared, etc.), before finalizing the fix I'd like to ask for the expert's input if adding -extldflags is an option? Thanks.

[ianlancetaylor]

@shawn-xdji This issue was closed several months ago, so this is not the best place to discuss possible changes. Please open a new issue or send a message to golang-dev. Thanks.