lld 13+ stopped working with the linker flag --gc-sections and Swift on ELF platforms

[finagolfin]

Describe the bug
SPM turned on --gc-sections in trunk six months ago, swiftlang/swift-package-manager#4135- just before that, a linux user reported this bug with lld 13 but didn't respond when I asked for more info- and others soon reported similar problems with SPM and lld. The issue appears to be a change to how --gc-sections works in lld that was added last year, which was then flipped on by default and produces linker errors like these when the repro command below is run:

Swift version 5.6.2 (swift-5.6.2-RELEASE)
Target: x86_64-unknown-linux-gnu
/home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/bin/swift-frontend -frontend -c -primary-file swift/test/Interpreter/hello_toplevel.swift -target x86_64-unknown-linux-gnu -disable-objc-interop -new-driver-path /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/bin/swift-driver -resource-dir /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift -module-name hello_toplevel -o /tmp/TemporaryDirectory.s5r53T/hello_toplevel-1.o
/home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/bin/swift-autolink-extract /tmp/TemporaryDirectory.s5r53T/hello_toplevel-1.o -o /tmp/TemporaryDirectory.s5r53T/hello_toplevel-2.autolink
/home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/bin/clang -fuse-ld=lld -pie -Xlinker -rpath -Xlinker /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o /tmp/TemporaryDirectory.s5r53T/hello_toplevel-1.o @/tmp/TemporaryDirectory.s5r53T/hello_toplevel-2.autolink -L /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux -lswiftCore --target=x86_64-unknown-linux-gnu -v -Xlinker --gc-sections -o hello_toplevel
error: link command failed with exit code 1 (use -v to see invocation)
clang version 13.0.0 (https://github.com/apple/llvm-project.git f765bf5b71fd3637a6f6d1d3e6ab95ca91892a0c)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/bin/ld.lld" -pie -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o hello_toplevel /lib/x86_64-linux-gnu/Scrt1.o /lib/x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/9/crtbeginS.o -L/home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux -L/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/bin/../lib -L/lib -L/usr/lib -rpath /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o /tmp/TemporaryDirectory.s5r53T/hello_toplevel-1.o -lswift_Concurrency -lswiftCore -lswiftSwiftOnoneSupport -lswiftCore --gc-sections -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/9/crtendS.o /lib/x86_64-linux-gnu/crtn.o
ld.lld: error: undefined hidden symbol: __start_swift5_protocols
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __stop_swift5_protocols
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __start_swift5_protocol_conformances
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __stop_swift5_protocol_conformances
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __start_swift5_type_metadata
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __stop_swift5_type_metadata
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __start_swift5_typeref
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __stop_swift5_typeref
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __start_swift5_reflstr
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __stop_swift5_reflstr
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __start_swift5_fieldmd
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __stop_swift5_fieldmd
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __start_swift5_assocty
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __stop_swift5_assocty
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __start_swift5_replace
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __stop_swift5_replace
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __start_swift5_replac2
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __stop_swift5_replac2
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __start_swift5_builtin
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: undefined hidden symbol: __stop_swift5_builtin
>>> referenced by SwiftRT-ELF.cpp
>>>               /home/foo/swift-5.6.2-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o:(swift_image_constructor())

ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

Steps To Reproduce
Steps to reproduce the behavior:

1. ./swift-5.6.2-RELEASE-ubuntu20.04/usr/bin/swiftc swift/test/Interpreter/hello_toplevel.swift -use-ld=lld -Xlinker --gc-sections -v

Removing the lld flag makes it default to the system gold linker on linux, which works fine. Make sure you're running lld 13+ as in the output above, as I think it may pick up an earlier lld installed in the system if one is there.

Expected behavior
Linking with lld 13+ and --gc-sections to work

Environment (please fill out the following information)

• OS: Ubuntu 20.04 x86_64 and Android 12 AArch64

Additional context
Since linux uses gold by default and the Android LTS NDK used lld 12 till a couple weeks ago, it looks like this wasn't affecting too many people, but will if SPM 5.7 turns on dead-stripping by default, so that change is being backed out of SPM temporarily, swiftlang/swift-package-manager#5698, till we can fix it here.

@keith has proposed #60357 to fix this, but it may not be enough. @drodriguez ran that fix and the new lld flag -z nostart-stop-gc through the compiler validation suite on linux x86_64 and got a couple dozen more test failures.

I tried replicating what he did natively on Android with the July 25 source snapshot, which uses lld 14 from the Termux environment, and got the following compiler validation suite test results:

No changes, ie just the stock test run with lld - 36 test failures
lld with SWIFT_DRIVER_TEST_OPTIONS=" -Xlinker --gc-sections" - 1113 test failures
lld with SWIFT_DRIVER_TEST_OPTIONS=" -Xlinker --gc-sections" and the stdlib fix from #60357 - 159 test failures
lld with SWIFT_DRIVER_TEST_OPTIONS=" -Xlinker --gc-sections -Xlinker -z -Xlinker nostart-stop-gc" - 54 test failures

I also tried applying both the stdlib fix and -z nostart-stop-gc, but that made no difference in the test results compared to -z nostart-stop-gc alone.

Comparing that last run to the first stock run, I see 19 additional test failures- 18 TypeDecoder tests and DebugInfo.ASTSection- that all invoke lldb-moduleimport-test and fail with FileCheck error: '<stdin>' is empty., similar to @drodriguez's results (LinkerSections.function_sections-lld that failed in the stock run now passes with -z nostart-stop-gc applied, as that test was already using --gc-sections). I will look into those further.

I'm looking for feedback from the Swift compiler devs on what we should do about this: apply one of these fixes or try something different?

@3405691582, let me know if you can reproduce on OpenBSD.

[finagolfin]

I looked into those 19 lldb-moduleimport-test failures with nostart-stop-gc: they're all tests that add debug info and have that lldb executable check it, so it makes sense that they no longer work with --gc-sections stripping that info out. Since the tests will never run with this stripping enabled, these 19 tests can be ignored as unrelated to this issue.

Ideally, we'd pass nostart-stop-gc when linking using lld with --gc-sections and the best place to do that would be in SPM, since it is difficult for the toolchain to extract which linker is being used, but unfortunately when I added -Xlinker -z -Xlinker nostart-stop-gc to the above repro command and removed the -use-ld=lld flag, so it uses gold by default instead, linking fails because gold doesn't recognize -z nostart-stop-gc and errors out (I tried the original binutils ld and it doesn't recognize that flag either, but simply puts out a warning and still works).

So we may be stuck adding that linker flag here in the compiler itself after searching the passed in linker flags for --gc-sections, let me know if anyone has a better suggestion.

[3405691582]

Yes, I'm seemingly bit by this too. Ran in to this building Swift proper at HEAD, so it might not just be limited to SPM.

[finagolfin]

Ran in to this building Swift proper at HEAD, so it might not just be limited to SPM.

Hm, you're seeing --gc-sections added when building the stdlib? I don't think it should, so maybe that's coming from somewhere else. This problem is only with lld 13+ and that stripping flag.

[3405691582]

Yes. I need to look into it a little more closely later -- the mention of the "R" flag on the other bug was a reasonable workaround.

[finagolfin]

Alright, thanks for reporting that this ELF issue is hitting more platforms, would be good to know more about that issue when building this repo, which I haven't seen.

[keith]

If you use lld to build the swift compiler in general you hit this today when linking swiftrt.o, I found the same when I was debugging this issue and needed to use lld to avoid linking OOMs. I also found that R worked fine, but that will likely result in more issues later when using the produced swiftrt.o. I don't know where that flag is coming from but I'm also not super surprised that something is adding it, if someone can find where we can add the -z nostart-stop-gc there at least to avoid this issue for folks working on the compiler.

[finagolfin]

If you use lld to build the swift compiler in general you hit this today when linking swiftrt.o, I found the same when I was debugging this issue and needed to use lld to avoid linking OOMs.

Huh, I switched over to using lld for Android ever since NDK 23 last year threw out binutils, #39045, and haven't seen that. I only see problems when turning on --gc-sections also. I wonder if that flag is getting turned on externally for you two somehow, or if you're seeing other problems with lld even without that stripping flag.

if someone can find where we can add the -z nostart-stop-gc there at least to avoid this issue for folks working on the compiler.

Since only lld supports this flag, I want to add it only if lld is passed in to -use-ld=, but I was unaware that flag took paths too until you mentioned it, so that will take a bit more looking into. I just checked and lld will accept -z nostart-stop-gc and just ignore it if --gc-sections is not passed in, so I will just add the former for lld regardless of the latter stripping flag being passed in or not. I'll submit a pull soon.

Of course, that will break using lld 12 and earlier, since they don't know the nostart-stop-gc flag and error, but it's been almost a year since lld 13 came out, so should be fine. As for people setting their system linker to lld, unless there's some way to detect that similar to how SPM always gets the host triple from the Swift compiler it's using, swiftlang/swift-package-manager#2519, we can't know what linker they're using, so they're on their own. The good news is that lld 14 now puts out more info about this issue in the error message, so that should clue them in.

[3405691582]

Another minor update/jotting some notes down: in base Swift, it's the lib_InternalSwiftSyntaxParser.so link that fails because of this, and includes -Wl,--gc-sections because add_link_opts(libSwiftSyntaxParser) and add_link_opts comes from AddLLVM.cmake in some shape or form.

[finagolfin]

Oh yeah, I've hit that before and had to remove that flag for libSwiftSyntaxParser for a trunk snapshot or two, then the problem just went away on its own. 🤷

[keith]

reopening to track the core issue

[keith]

Relevant testing info for this issue here: swiftlang/swift-package-manager#5698 (comment)

[keith]

I can no longer repro issue with the Reflection tests that we were previously discussing.

[keith]

For reference: https://lld.llvm.org/ELF/start-stop-gc

[finagolfin]

This was fixed by #72061 and associated changes, which will ship in the upcoming Swift 6.