MIPS host bootstrap: compiler_rt stack overflows #37823

xen0n · 2016-11-17T08:01:32Z

The cross-compiled MIPS host compilers are fine, but stage1 immediately segfaults when asked to do just about anything:

// x.rs
fn main() {
}

$ ./build/mips64el-unknown-linux-gnuabi64/stage1/bin/rustc ./x.rs

thread 'rustc' has overflowed its stack
fatal runtime error: stack overflow
[1]    3433 IOT instruction (core dumped)  ./build/mips64el-unknown-linux-gnuabi64/stage1/bin/rustc ./x.rs

(gdb) r ./x.rs
Starting program: /opt/store/src/rust/build/mips64el-unknown-linux-gnuabi64/stage1/bin/rustc ./x.rs
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0xfff353ef10 (LWP 3476)]

Thread 2 "rustc" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xfff353ef10 (LWP 3476)]
0x000000fff57198b8 in __ctzdi2 () from /opt/store/src/rust/build/mips64el-unknown-linux-gnuabi64/stage1/bin/../lib/../lib/librustc_llvm-d0f44aedc4c18d77.so
(gdb) disas
Dump of assembler code for function __ctzdi2:
   0x000000fff57198b0 <+0>:     daddiu  sp,sp,-32
   0x000000fff57198b4 <+4>:     sll     v0,a0,0x0
=> 0x000000fff57198b8 <+8>:     sd      s0,8(sp)
   0x000000fff57198bc <+12>:    sltiu   s0,v0,1
   0x000000fff57198c0 <+16>:    sd      gp,16(sp)
   0x000000fff57198c4 <+20>:    dnegu   s0,s0
   0x000000fff57198c8 <+24>:    lui     gp,0x96
   0x000000fff57198cc <+28>:    nor     a1,zero,s0
   0x000000fff57198d0 <+32>:    daddu   gp,gp,t9
   0x000000fff57198d4 <+36>:    dsra32  a0,a0,0x0
   0x000000fff57198d8 <+40>:    and     v1,a0,s0
   0x000000fff57198dc <+44>:    daddiu  gp,gp,18760
   0x000000fff57198e0 <+48>:    and     a0,a1,v0
   0x000000fff57198e4 <+52>:    ld      t9,-32584(gp)
   0x000000fff57198e8 <+56>:    or      a0,v1,a0
   0x000000fff57198ec <+60>:    dsll32  a0,a0,0x0
   0x000000fff57198f0 <+64>:    sd      ra,24(sp)
   0x000000fff57198f4 <+68>:    bal     0xfff57198b0 <__ctzdi2>
   0x000000fff57198f8 <+72>:    dsrl32  a0,a0,0x0
   0x000000fff57198fc <+76>:    ld      ra,24(sp)
   0x000000fff5719900 <+80>:    andi    s0,s0,0x20
   0x000000fff5719904 <+84>:    addu    v0,v0,s0
   0x000000fff5719908 <+88>:    ld      gp,16(sp)
   0x000000fff571990c <+92>:    ld      s0,8(sp)
   0x000000fff5719910 <+96>:    jr      ra
   0x000000fff5719914 <+100>:   daddiu  sp,sp,32
End of assembler dump.

Notice the recursion, which is obviously wrong, and non-existent on stage0:

$ ar x ../../../nightly/2016-11-14/lib/rustlib/mips64el-unknown-linux-gnuabi64/lib/libcompiler_builtins-e428224f6caf212a.rlib
$ objdump -d ctzdi2.o
ctzdi2.o:     file format elf64-tradlittlemips


Disassembly of section .text.__ctzdi2:

0000000000000000 <__ctzdi2>:
   0:   00041800        sll     v1,a0,0x0
   4:   2c650001        sltiu   a1,v1,1
   8:   0005282f        dnegu   a1,a1
   c:   0004203f        dsra32  a0,a0,0x0
  10:   0065200a        movz    a0,v1,a1
  14:   00041023        negu    v0,a0
  18:   00822024        and     a0,a0,v0
  1c:   70842020        clz     a0,a0
  20:   2402001f        li      v0,31
  24:   00441023        subu    v0,v0,a0
  28:   30a50020        andi    a1,a1,0x20
  2c:   03e00008        jr      ra
  30:   00a21021        addu    v0,a1,v0
  34:   00000000        nop

This is LLVM bug 11663. Maybe we should incorporate the workaround there as well.

The text was updated successfully, but these errors were encountered:

xen0n · 2016-11-17T08:23:48Z

This FreeBSD change seems more complete than the one found in the LLVM bug thread.

alexcrichton · 2016-11-17T15:38:57Z

It looks like the binaries we're distributing don't have this recursion, so I guess this is only happening when you're bootstrapping locally? If so, what compiler are you using?

It'd be useful to drill into the command that actually compiled ctzdi2.c so we can figure out what's going on here.

xen0n · 2016-11-17T16:11:48Z

I'm indeed bootstrapping natively on MIPS64el.

gcc --version -v output:

Using built-in specs.
COLLECT_GCC=/usr/mips64el-unknown-linux-gnu/gcc-bin/5.4.0/gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/mips64el-unknown-linux-gnu/5.4.0/lto-wrapper
gcc (Gentoo 5.4.0 p1.0, pie-0.6.5) 5.4.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


Target: mips64el-unknown-linux-gnu
Configured with: /var/tmp/portage/sys-devel/gcc-5.4.0/work/gcc-5.4.0/configure --host=mips64el-unknown-linux-gnu --build=mips64el-unknown-linux-gnu --prefix=/usr --bindir=/usr/mips64el-unknown-linux-gnu/gcc-bin/5.4.0 --includedir=/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/include --datadir=/usr/share/gcc-data/mips64el-unknown-linux-gnu/5.4.0 --mandir=/usr/share/gcc-data/mips64el-unknown-linux-gnu/5.4.0/man --infodir=/usr/share/gcc-data/mips64el-unknown-linux-gnu/5.4.0/info --with-gxx-include-dir=/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/include/g++-v5 --with-python-dir=/share/gcc-data/mips64el-unknown-linux-gnu/5.4.0/python --enable-languages=c,c++,fortran --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --disable-nls --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 5.4.0 p1.0, pie-0.6.5' --enable-libstdcxx-time --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --enable-multilib --disable-altivec --disable-fixed-point --with-abi=64 --disable-libgcj --enable-libgomp --disable-libmudflap --disable-libssp --disable-libcilkrts --disable-libmpx --disable-vtable-verify --disable-libvtv --enable-lto --with-isl --disable-isl-version-check --disable-libsanitizer
Thread model: posix
gcc version 5.4.0 (Gentoo 5.4.0 p1.0, pie-0.6.5) 
COLLECT_GCC_OPTIONS='--version' '-v' '-mabi=64' '-mllsc' '-mno-shared' '-EL'
 /usr/libexec/gcc/mips64el-unknown-linux-gnu/5.4.0/cc1 -quiet -v help-dummy -mel -quiet -dumpbase help-dummy -mabi=64 -mllsc -mno-shared -auxbase help-dummy -version --version -fstack-protector-strong -o /tmp/ccS1TELP.s
GNU C11 (Gentoo 5.4.0 p1.0, pie-0.6.5) version 5.4.0 (mips64el-unknown-linux-gnu)
    compiled by GNU C version 5.4.0, GMP version 6.1.1, MPFR version 3.1.4, MPC version 1.0.3
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
COLLECT_GCC_OPTIONS='--version' '-v' '-mabi=64' '-mllsc' '-mno-shared' '-EL'
 /usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../../mips64el-unknown-linux-gnu/bin/as -v -EL -O1 -no-mdebug -mabi=64 -mno-shared -KPIC --version -o /tmp/cciWbyMv.o /tmp/ccS1TELP.s
GNU assembler version 2.26.1 (mips64el-unknown-linux-gnu) using BFD version (Gentoo 2.26.1 p1.0) 2.26.1
GNU assembler (Gentoo 2.26.1 p1.0) 2.26.1
Copyright (C) 2015 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `mips64el-unknown-linux-gnu'.
COMPILER_PATH=/usr/libexec/gcc/mips64el-unknown-linux-gnu/5.4.0/:/usr/libexec/gcc/mips64el-unknown-linux-gnu/5.4.0/:/usr/libexec/gcc/mips64el-unknown-linux-gnu/:/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/:/usr/lib/gcc/mips64el-unknown-linux-gnu/:/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../../mips64el-unknown-linux-gnu/bin/
LIBRARY_PATH=/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/:/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../../lib64/:/lib64/../lib64/:/usr/lib64/../lib64/:/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../../mips64el-unknown-linux-gnu/lib/:/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../:/lib64/:/usr/lib64/
COLLECT_GCC_OPTIONS='--version' '-v' '-mabi=64' '-mllsc' '-mno-shared' '-EL'
 /usr/libexec/gcc/mips64el-unknown-linux-gnu/5.4.0/collect2 -plugin /usr/libexec/gcc/mips64el-unknown-linux-gnu/5.4.0/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/mips64el-unknown-linux-gnu/5.4.0/lto-wrapper -plugin-opt=-fresolution=/tmp/ccatiVca.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --eh-frame-hdr -EL -dynamic-linker /lib64/ld.so.1 -melf64ltsmip --version /usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../../lib64/crt1.o /usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../../lib64/crti.o /usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/crtbegin.o -L/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0 -L/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../../lib64 -L/lib64/../lib64 -L/usr/lib64/../lib64 -L/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../../mips64el-unknown-linux-gnu/lib -L/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../.. -L/lib64 -L/usr/lib64 /tmp/cciWbyMv.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/crtend.o /usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../../lib64/crtn.o
collect2 version 5.4.0
/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../../mips64el-unknown-linux-gnu/bin/ld -plugin /usr/libexec/gcc/mips64el-unknown-linux-gnu/5.4.0/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/mips64el-unknown-linux-gnu/5.4.0/lto-wrapper -plugin-opt=-fresolution=/tmp/ccatiVca.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --eh-frame-hdr -EL -dynamic-linker /lib64/ld.so.1 -melf64ltsmip --version /usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../../lib64/crt1.o /usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../../lib64/crti.o /usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/crtbegin.o -L/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0 -L/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../../lib64 -L/lib64/../lib64 -L/usr/lib64/../lib64 -L/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../../mips64el-unknown-linux-gnu/lib -L/usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../.. -L/lib64 -L/usr/lib64 /tmp/cciWbyMv.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/crtend.o /usr/lib/gcc/mips64el-unknown-linux-gnu/5.4.0/../../../../lib64/crtn.o
GNU ld (Gentoo 2.26.1 p1.0) 2.26.1
Copyright (C) 2015 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

Rust revision used is c87bae6 with #37800 applied. I'll try to look into the actual compiler invocation, as even though I applied the workaround the resulting __ctzdi2 still is broken.

xen0n · 2016-11-17T17:40:13Z

The compiler invocation is like this (newlines added by me):

cc -O2 -ffunction-sections -fdata-sections -ffunction-sections -fdata-sections -fPIC -fPIC \
-fno-builtin -fvisibility=hidden -fomit-frame-pointer -ffreestanding \
-o /opt/store/src/rust/build/mips64el-unknown-linux-gnuabi64/stage0-std/mips64el-unknown-linux-gnuabi64/release/build/compiler_builtins-5574dec8ddf5551f/out/../compiler-rt/lib/builtins/ctzdi2.o \
-c ../compiler-rt/lib/builtins/ctzdi2.c

Ignore my previous comment that the workaround was ignored, I forgot to commit the submodule update and rustbuild reverted it to a previously broken one. The generated code now looks like this:

ctzdi2.o:     file format elf64-tradlittlemips


Disassembly of section .text:

0000000000000000 <__ctzdi2>:
   0:   67bdffe0        daddiu  sp,sp,-32
   4:   00041000        sll     v0,a0,0x0
   8:   ffbc0010        sd      gp,16(sp)
   c:   3c1c0000        lui     gp,0x0
  10:   ffb00008        sd      s0,8(sp)
  14:   0399e02d        daddu   gp,gp,t9
  18:   2c500001        sltiu   s0,v0,1
  1c:   679c0000        daddiu  gp,gp,0
  20:   0010802f        dnegu   s0,s0
  24:   00102827        nor     a1,zero,s0
  28:   df990000        ld      t9,0(gp)
  2c:   0004203f        dsra32  a0,a0,0x0
  30:   00901824        and     v1,a0,s0
  34:   00a22024        and     a0,a1,v0
  38:   ffbf0018        sd      ra,24(sp)
  3c:   0320f809        jalr    t9
  40:   00642025        or      a0,v1,a0
  44:   dfbf0018        ld      ra,24(sp)
  48:   32100020        andi    s0,s0,0x20
  4c:   02021021        addu    v0,s0,v0
  50:   dfbc0010        ld      gp,16(sp)
  54:   dfb00008        ld      s0,8(sp)
  58:   03e00008        jr      ra
  5c:   67bd0020        daddiu  sp,sp,32

But the function called is now __ctzsi2 (not in the output; MIPS objdump seems feature-lacking). I kicked off compilation now, hopefully stage1 would be done several hours later (the hardware is slow). I'll run some tests to see if the workaround really worked.

alexcrichton · 2016-11-17T17:56:27Z

Hm ok, so is the same bug still happening? Or we're waiting to see if it's a new bug?

Also, what was the workaround you applied? (e.g. the patch)

xen0n · 2016-11-17T18:29:48Z

The commit is xen0n/compiler-rt@556504d. I just saw a different crash but it's probably unrelated and spurious, I'm restarting the build to see if that's the case.

Building stage0 compiler artifacts (mips64el-unknown-linux-gnuabi64 -> mips64el-unknown-linux-gnuabi64)
   Compiling arena v0.0.0 (file:///opt/store/src/rust/src/libarena)
   Compiling log v0.0.0 (file:///opt/store/src/rust/src/liblog)
   Compiling rustc_bitflags v0.0.0 (file:///opt/store/src/rust/src/librustc_bitflags)
   Compiling graphviz v0.0.0 (file:///opt/store/src/rust/src/libgraphviz)
   Compiling rustdoc v0.0.0 (file:///opt/store/src/rust/src/librustdoc)
   Compiling rustc_llvm v0.0.0 (file:///opt/store/src/rust/src/librustc_llvm)
   Compiling flate v0.0.0 (file:///opt/store/src/rust/src/libflate)
   Compiling fmt_macros v0.0.0 (file:///opt/store/src/rust/src/libfmt_macros)
   Compiling serialize v0.0.0 (file:///opt/store/src/rust/src/libserialize)
   Compiling rustc_platform_intrinsics v0.0.0 (file:///opt/store/src/rust/src/librustc_platform_intrinsics)
   Compiling rustc_data_structures v0.0.0 (file:///opt/store/src/rust/src/librustc_data_structures)
   Compiling syntax_pos v0.0.0 (file:///opt/store/src/rust/src/libsyntax_pos)
   Compiling rustc_errors v0.0.0 (file:///opt/store/src/rust/src/librustc_errors)
   Compiling syntax v0.0.0 (file:///opt/store/src/rust/src/libsyntax)
   Compiling rustc_const_math v0.0.0 (file:///opt/store/src/rust/src/librustc_const_math)
   Compiling rustc_back v0.0.0 (file:///opt/store/src/rust/src/librustc_back)
   Compiling proc_macro_tokens v0.0.0 (file:///opt/store/src/rust/src/libproc_macro_tokens)
   Compiling proc_macro v0.0.0 (file:///opt/store/src/rust/src/libproc_macro)
   Compiling syntax_ext v0.0.0 (file:///opt/store/src/rust/src/libsyntax_ext)
   Compiling rustc v0.0.0 (file:///opt/store/src/rust/src/librustc)
*** Error in `/opt/store/src/rust/build/mips64el-unknown-linux-gnuabi64/stage0/bin/rustc': double free or corruption (out): 0x000000ffe155dea0 ***
error: Could not compile `rustc`.

To learn more, run the command again with --verbose.


command did not execute successfully: "/opt/store/src/rust/build/mips64el-unknown-linux-gnuabi64/stage0/bin/cargo" "build" "-j" "4" "--target" "mips64el-unknown-linux-gnuabi64" "--release" "-
-features" " jemalloc" "--manifest-path" "/opt/store/src/rust/src/rustc/Cargo.toml"
expected success, got: exit code: 101

xen0n · 2016-11-17T18:30:49Z

As for the ctzdi2 bug: it is presumably fixed by the workaround. But we can never be sure until we've run the tests.

xen0n · 2016-11-17T18:58:07Z

Quick update: the double free was indeed spurious, the build is happily running now. Unfortunately I can't test the stage1 immediately, it's 3 am here. I'll see if the build can progress to anything more than stage1 later in the day.

alexcrichton · 2016-11-17T21:25:11Z

If you want to send that patch to our compiler-rt fork, I'd be happy to merge!

xen0n · 2016-11-18T04:49:33Z

Problem solved, stage2 finished this morning and test failures are few, currently there are

    [compile-fail] compile-fail/allocator-rust-dylib-is-jemalloc.rs
    [compile-fail] compile-fail/asm-bad-clobber.rs
    [compile-fail] compile-fail/asm-in-bad-modifier.rs
    [compile-fail] compile-fail/asm-misplaced-option.rs
    [compile-fail] compile-fail/asm-out-assign-imm.rs
    [compile-fail] compile-fail/asm-out-no-modifier.rs
    [compile-fail] compile-fail/asm-out-read-uninit.rs

which are all expected to fail on MIPS but not ignored. I'll prepare a PR to compile-rt shortly.

xen0n · 2016-11-18T05:19:29Z

I've opened rust-lang/compiler-rt#27. I'll open another PR to update the submodule after that PR is merged.

Fixes rust-lang#37823.

@alexcrichton

Update compiler-rt to fix MIPS64 infinite recursion Fixes #37823. Test fixes are coming in a separate PR later. r? @alexcrichton

xen0n changed the title ~~MIPS host bootstrap: compiler_rt: ABI broken, stack overflows~~ MIPS host bootstrap: compiler_rt stack overflows Nov 17, 2016

xen0n mentioned this issue Nov 18, 2016

Update compiler-rt to fix MIPS64 infinite recursion #37849

Merged

xen0n added a commit to xen0n/rust that referenced this issue Nov 21, 2016

Update compiler-rt to fix MIPS64 infinite recursion

d37c025

Fixes rust-lang#37823.

bors added a commit that referenced this issue Nov 24, 2016

Auto merge of #37849 - xen0n:compiler-rt-bump, r=alexcrichton

696fab8

Update compiler-rt to fix MIPS64 infinite recursion Fixes #37823. Test fixes are coming in a separate PR later. r? @alexcrichton

bors closed this as completed in #37849 Nov 24, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MIPS host bootstrap: compiler_rt stack overflows #37823

MIPS host bootstrap: compiler_rt stack overflows #37823

xen0n commented Nov 17, 2016 •

edited

Loading

xen0n commented Nov 17, 2016

alexcrichton commented Nov 17, 2016

xen0n commented Nov 17, 2016

xen0n commented Nov 17, 2016

alexcrichton commented Nov 17, 2016

xen0n commented Nov 17, 2016

xen0n commented Nov 17, 2016

xen0n commented Nov 17, 2016

alexcrichton commented Nov 17, 2016

xen0n commented Nov 18, 2016

xen0n commented Nov 18, 2016

MIPS host bootstrap: compiler_rt stack overflows #37823

MIPS host bootstrap: compiler_rt stack overflows #37823

Comments

xen0n commented Nov 17, 2016 • edited Loading

xen0n commented Nov 17, 2016

alexcrichton commented Nov 17, 2016

xen0n commented Nov 17, 2016

xen0n commented Nov 17, 2016

alexcrichton commented Nov 17, 2016

xen0n commented Nov 17, 2016

xen0n commented Nov 17, 2016

xen0n commented Nov 17, 2016

alexcrichton commented Nov 17, 2016

xen0n commented Nov 18, 2016

xen0n commented Nov 18, 2016

xen0n commented Nov 17, 2016 •

edited

Loading