Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GlobalAlloc trait + tweaks for initial stabilization #49669

Merged
merged 27 commits into from
Apr 13, 2018

Conversation

SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Apr 4, 2018

This is the outcome of discussion at the Rust All Hands in Berlin. The high-level goal is stabilizing sooner rather than later the ability to change the global allocator, as well as allocating memory without abusing Vec::with_capacity + mem::forget.

Since we’re not ready to settle every detail of the Alloc trait for the purpose of collections that are generic over the allocator type (for example the possibility of a separate trait for deallocation only, and what that would look like exactly), we propose introducing separately a new GlobalAlloc trait, for use with the #[global_allocator] attribute.

We also propose a number of changes to existing APIs. They are batched in this one PR in order to minimize disruption to Nightly users.

The plan for initial stabilization is detailed in the tracking issue #49668.

CC @rust-lang/libs, @glandium

Immediate breaking changes to unstable features

  • For pointers to allocated memory, change the pointed type from u8 to Opaque, a new public extern type. Since extern types are not Sized, <*mut _>::offset cannot be used without first casting to another pointer type. (We hope that extern types can also be stabilized soon.)
  • In the Alloc trait, change these pointers to ptr::NonNull and change the AllocErr type to a zero-size struct. This makes return types Result<ptr::NonNull<Opaque>, AllocErr> be pointer-sized.
  • Instead of a new Layout, realloc takes only a new size (in addition to the pointer and old Layout). Changing the alignment is not supported with realloc.
  • Change the return type of Layout::from_size_align from Option<Self> to Result<Self, LayoutErr>, with LayoutErr a new opaque struct.
  • A static item registered as the global allocator with the #[global_allocator] must now implement the new GlobalAlloc trait instead of Alloc.

Eventually-breaking changes to unstable features, with a deprecation period

  • Rename the respective heap modules to alloc in the core, alloc, and std crates. (Yes, this does mean that ::alloc::alloc::Alloc::alloc is a valid path to a trait method if you have exetrn crate alloc;)
  • Rename the the Heap type to Global, since it is the entry point for what’s registered with #[global_allocator].

Old names remain available for now, as deprecated pub use reexports.

Backward-compatible changes

  • Add a new extern type Opaque, for use in pointers to allocated memory.
  • Add a new GlobalAlloc trait shown below. Unlike Alloc, it uses bare *mut Opaque without NonNull or Result. NULL in return values indicates an error (of unspecified nature). This is easier to implement on top of malloc-like APIs.
  • Add impls of GlobalAlloc for both the Global and System types, in addition to existing impls of Alloc. This enables calling GlobalAlloc methods on the stable channel before Alloc is stable. Implementing two traits with identical method names can make some calls ambiguous, but most code is expected to have no more than one of the two traits in scope. Erroneous code like use std::alloc::Global; #[global_allocator] static A: Global = Global; (where Global is defined to call itself, causing infinite recursion) is not statically prevented by the type system, but we count on it being hard enough to do accidentally and easy enough to diagnose.
extern {
    pub type Opaque;
}

pub unsafe trait GlobalAlloc {
    unsafe fn alloc(&self, layout: Layout) -> *mut Opaque;
    unsafe fn dealloc(&self, ptr: *mut Opaque, layout: Layout);

    unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut Opaque { 
        // Default impl: self.alloc() and ptr::write_bytes()
    }
    unsafe fn realloc(&self, ptr: *mut Opaque, old_layout: Layout, new_size: usize) -> *mut Opaque {
        // Default impl: self.alloc() and ptr::copy_nonoverlapping() and self.dealloc()
    }

    fn oom(&self) -> ! {
        // intrinsics::abort
    }
    
    // More methods with default impls may be added in the future
}

Bikeshed

The tracking issue #49668 lists some open questions. If consensus is reached before this PR is merged, changes can be integrated.

@TimNN
Copy link
Contributor

TimNN commented Apr 4, 2018

Your PR failed on Travis. Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
Receiving objects: 100% (304/304), 48.86 KiB | 24.43 MiB/s, done.
---
Resolving deltas: 100% (247/247), completed with 74 local objects.
---
[00:00:00] Attempting with retry: sh -c rm -f download-src-llvm.tar.gz &&         curl -sSL -o download-src-llvm.tar.gz https://github.com/rust-lang/llvm/archive/7243155b1c3da0a980c868a87adebf00e0b33989.tar.gz
---
[00:00:48] configure: rust.quiet-tests     := True
---
[00:04:11] * 179 features
[00:04:11] tidy error: /checkout/src/liballoc/oom.rs:19: platform-specific cfg: cfg(any(unix, target_os = "redox"))
[00:04:11] tidy error: /checkout/src/liballoc/oom.rs:67: platform-specific cfg: cfg(not(any(windows, unix, target_os = "redox")))
[00:04:12] some tidy checks failed
[00:04:12]
[00:04:12]
[00:04:12] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:04:12] expected success, got: exit code: 1
[00:04:12]
[00:04:12]
[00:04:12] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:04:12] Build completed unsuccessfully in 0:01:44
[00:04:12] Makefile:79: recipe for target 'tidy' failed
[00:04:12] make: *** [tidy] Error 1
---
$ ls -lat $HOME/Library/Logs/DiagnosticReports/
ls: cannot access /home/travis/Library/Logs/DiagnosticReports/: No such file or directory
travis_time:end:00bac534:start=1522878325536986682,finish=1522878325544062106,duration=7075424
travis_fold:end:after_failure.2
travis_fold:start:after_failure.3
travis_time:start:28f99d88
$ find $HOME/Library/Logs/DiagnosticReports -type f -name '*.crash' -not -name '*.stage2-*.crash' -not -name 'com.apple.CoreSimulator.CoreSimulatorService-*.crash' -exec printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" {} \; -exec head -750 {} \; -exec echo travis_fold":"end:crashlog \; || true
find: `/home/travis/Library/Logs/DiagnosticReports': No such file or directory
travis_time:end:28f99d88:start=1522878325550815862,finish=1522878325559702359,duration=8886497
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:1d0ad1ac
$ dmesg | grep -i kill
[   10.868621] init: failsafe main process (1093) killed by TERM signal

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN.

@SimonSapin SimonSapin added A-allocators Area: Custom and system allocators T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. labels Apr 4, 2018
@SimonSapin
Copy link
Contributor Author

tidy error: /home/simon/rust1/src/liballoc/oom.rs:19: platform-specific cfg: cfg(any(unix, target_os = "redox"))
tidy error: /home/simon/rust1/src/liballoc/oom.rs:67: platform-specific cfg: cfg(not(any(windows, unix, target_os = "redox")))

I moved this cfg-using code from liballoc_system to liballoc in order to implement <Global as Alloc>::oom (which is used by code in liballoc that allocates) because the GlobalAlloc trait does not have a oom method, so it cannot be forwarded "through" #[global_allocator].

@SimonSapin
Copy link
Contributor Author

I’ve added an exception in tidy.

@@ -756,7 +753,7 @@ mod tests {
// before allocation attempts start failing.
struct BoundedAlloc { fuel: usize }
unsafe impl Alloc for BoundedAlloc {
unsafe fn alloc(&mut self, layout: Layout) -> Result<*mut u8, AllocErr> {
unsafe fn alloc(&mut self, layout: Layout) -> Result<NonNull<u8>, AllocErr> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have NonNull<u8> leftovers in this file.

@@ -737,7 +737,7 @@ impl<T: Clone> RcFromSlice<T> for Rc<[T]> {
// In the event of a panic, elements that have been written
// into the new RcBox will be dropped, then the memory freed.
struct Guard<T> {
mem: *mut u8,
mem: NonNull<u8>,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could use NonNull<Void> here too.

@@ -9,6 +9,7 @@ path = "lib.rs"

[dependencies]
core = { path = "../libcore" }
libc = { path = "../rustc/libc_shim" }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

liballoc should not depend on libc. This will break code that runs in kernel-like environments where you have a global memory allocator but no libc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if libc was only a dependency for cfg(any(unix, target_os = "redox"))? It’s only used there.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer if liballoc was completely OS-independent. I am currently using Rust to make Linux binaries which do not link to libc (just using raw syscalls).

This is why, when I originally implemented printing a message on OOM (#30801), I allowed the OOM to be dynamically set at runtime. It would default to intrinsics::abort, but would be overridden by libstd on startup to a handler that prints a message.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you’re saying that a oom method should be added to the GlobalAlloc trait so that #[global_allocator] can forward it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking of adding back a dynamic OOM handler, but your idea is better.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does dynamic handler mean, if not the same as in my previous comment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never mind, I see later in the thread that you mention set_oom_handler.

}
}

/// The `CannotReallocInPlace` error is used when `grow_in_place` or
/// The `CannotgInPlace` error is used when `grow_in_place` or
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, thanks.

@TimNN
Copy link
Contributor

TimNN commented Apr 4, 2018

Your PR failed on Travis. Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
Receiving objects: 100% (311/311), 49.40 KiB | 24.70 MiB/s, done.
---
Resolving deltas: 100% (252/252), completed with 77 local objects.
---
[00:00:00] Attempting with retry: sh -c rm -f download-src-llvm.tar.gz &&         curl -sSL -o download-src-llvm.tar.gz https://github.com/rust-lang/llvm/archive/7243155b1c3da0a980c868a87adebf00e0b33989.tar.gz
---
[00:00:49] configure: rust.quiet-tests     := True
---
[00:41:44] ..........................................................................i.........................
[00:41:50] .................i..................................................................................
---
[00:42:27] ............................................................................................i.......
[00:42:34] ................................................................i...................................
---
[00:43:13] .............F......................................................................................
[00:43:29] .............................................i......................................................
---
[00:47:21] .............................i......................................................................
[00:47:35] ..............................................................i.....................................
[00:47:51] ...............................................i....................................................
[00:48:12] ....................................................................................................
[00:48:34] ....................................................................................................
[00:48:56] ....................................................................................................
[00:49:22] .i...............................................................................................i..
[00:49:48] .................................................................................test [run-pass] run-pass/mir_heavy_promoted.rs has been running for over 60 seconds
[00:49:59] ...................
[00:50:30] ..................................................F.................................................
[00:51:07] .............................................................ii.....................................
[00:51:52] ........................i....................................................i.ii....test [run-pass] run-pass/saturating-float-casts.rs has been running for over 60 seconds
[00:51:59] ...............
[00:52:40] .....................................................................................iiiiiii........
---
[00:54:17] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/run-pass/allocator/xcrate-use2.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass" "--target=x86_64-unknown-linux-gnu" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass/allocator/xcrate-use2.stage2-x86_64-unknown-linux-gnu" "-Crpath" "-O" "-Zmiri" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass/allocator/xcrate-use2.stage2-x86_64-unknown-linux-gnu.aux"
---
[00:54:17] error[E0599]: no method named `unwrap` found for type `*mut std::alloc::Void` in the current scope
[00:54:17]   --> /checkout/src/test/run-pass/allocator/xcrate-use2.rs:33:48
[00:54:17]    |
[00:54:17] 33 |         let ptr = Global.alloc(layout.clone()).unwrap();
[00:54:17]    |                                                ^^^^^^
[00:54:17]
[00:54:17] error[E0277]: the trait bound `std::fmt::Debug: std::marker::Sized` is not satisfied
[00:54:17]   --> /checkout/src/test/run-pass/allocator/xcrate-use2.rs:33:13
[00:54:17]    |
[00:54:17] 33 |         let ptr = Global.alloc(layout.clone()).unwrap();
[00:54:17]    |             ^^^ `std::fmt::Debug` does not have a constant size known at compile-time
[00:54:17]    |
[00:54:17]    = help: the trait `std::marker::Sized` is not implemented for `std::fmt::Debug`
[00:54:17]    = note: all local variables must have a statically known size
[00:54:17]
[00:54:17] error[E0308]: mismatched types
[00:54:17]   --> /checkout/src/test/run-pass/allocator/xcrate-use2.rs:36:24
[00:54:17]    |
[00:54:17] 36 |         Global.dealloc(ptr, layout.clone());
[00:54:17]    |                        ^^^ expected *-ptr, found trait std::fmt::Debug
[00:54:17]    |
[00:54:17]    = note: expected type `*mut std::alloc::Void`
[00:54:17]               found type `std::fmt::Debug`
---
[00:54:17] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/run-pass/realloc-16687.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass" "--target=x86_64-unknown-linux-gnu" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass/realloc-16687.stage2-x86_64-unknown-linux-gnu" "-Crpath" "-O" "-Zmiri" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass/realloc-16687.stage2-x86_64-unknown-linux-gnu.aux"
---
[00:54:17] warning: use of deprecated item 'std::alloc::Heap': type renamed to `Global`
[00:54:17]   --> /checkout/src/test/run-pass/realloc-16687.rs:18:17
[00:54:17]    |
[00:54:17] 18 | use std::heap::{Heap, Alloc, Layout};
[00:54:17]    |                 ^^^^
[00:54:17]    |
[00:54:17]    = note: #[warn(deprecated)] on by default
[00:54:17]
[00:54:17] warning: use of deprecated item 'std::alloc::Heap': type renamed to `Global`
[00:54:17]   --> /checkout/src/test/run-pass/realloc-16687.rs:53:19
[00:54:17]    |
[00:54:17] 53 |         let ret = Heap.alloc(layout.clone()).unwrap_or_else(|_| Heap.oom());
[00:54:17]    |                   ^^^^
[00:54:17]
[00:54:17] warning: use of deprecated item 'std::alloc::Heap': type renamed to `Global`
[00:54:17]   --> /checkout/src/test/run-pass/realloc-16687.rs:53:65
[00:54:17]    |
[00:54:17] 53 |         let ret = Heap.alloc(layout.clone()).unwrap_or_else(|_| Heap.oom());
[00:54:17]    |                                                                 ^^^^
[00:54:17]
[00:54:17] warning: use of deprecated item 'std::alloc::Heap': type renamed to `Global`
[00:54:17]   --> /checkout/src/test/run-pass/realloc-16687.rs:67:9
[00:54:17]    |
[00:54:17] 67 |         Heap.dealloc(NonNull::new_unchecked(ptr), layout);
[00:54:17]    |         ^^^^
[00:54:17]
[00:54:17] warning: use of deprecated item 'std::alloc::Heap': type renamed to `Global`
[00:54:17]   --> /checkout/src/test/run-pass/realloc-16687.rs:75:19
[00:54:17]    |
[00:54:17] 75 |         let ret = Heap.realloc(NonNull::new_unchecked(ptr), old.clone(), new.clone())
[00:54:17]    |                   ^^^^
[00:54:17]
[00:54:17] warning: use of deprecated item 'std::alloc::Heap': type renamed to `Global`
[00:54:17]   --> /checkout/src/test/run-pass/realloc-16687.rs:76:33
[00:54:17]    |
[00:54:17] 76 |             .unwrap_or_else(|_| Heap.oom());
[00:54:17]    |                                 ^^^^
[00:54:17]
[00:54:17] error[E0308]: mismatched types
[00:54:17]   --> /checkout/src/test/run-pass/realloc-16687.rs:59:9
[00:54:17]    |
[00:54:17] 48 |     unsafe fn allocate(layout: Layout) -> *mut u8 {
[00:54:17]    |                                           ------- expected `*mut u8` because of return type
[00:54:17] ...
[00:54:17] 59 |         ret.as_ptr()
[00:54:17]    |         ^^^^^^^^^^^^ expected u8, found extern type `std::alloc::Void`
[00:54:17]    |
[00:54:17]    = note: expected type `*mut u8`
[00:54:17]               found type `*mut std::alloc::Void`
[00:54:17]
[00:54:17] error[E0308]: mismatched types
[00:54:17]   --> /checkout/src/test/run-pass/realloc-16687.rs:67:45
[00:54:17]    |
[00:54:17] 67 |         Heap.dealloc(NonNull::new_unchecked(ptr), layout);
[00:54:17]    |                                             ^^^ expected extern type `std::alloc::Void`, found u8
[00:54:17]    |
[00:54:17]    = note: expected type `*mut std::alloc::Void`
[00:54:17]               found type `*mut u8`
[00:54:17]
[00:54:17] error[E0308]: mismatched types
[00:54:17]   --> /checkout/src/test/run-pass/realloc-16687.rs:75:55
[00:54:17]    |
[00:54:17] 75 |         let ret = Heap.realloc(NonNull::new_unchecked(ptr), old.clone(), new.clone())
[00:54:17]    |                                                       ^^^ expected extern type `std::alloc::Void`, found u8
[00:54:17]    |
[00:54:17]    = note: expected type `*mut std::alloc::Void`
[00:54:17]               found type `*mut u8`
[00:54:17]
[00:54:17] error[E0308]: mismatched types
[00:54:17]   --> /checkout/src/test/run-pass/realloc-16687.rs:75:74
[00:54:17]    |
[00:54:17] 75 |         let ret = Heap.realloc(NonNull::new_unchecked(ptr), old.clone(), new.clone())
[00:54:17]    |                                                                          ^^^^^^^^^^^ expected usize, found struct `std::alloc::Layout`
[00:54:17]    |
[00:54:17]    = note: expected type `usize`
[00:54:17]               found type `std::alloc::Layout`
[00:54:17]
[00:54:17] error[E0308]: mismatched types
[00:54:17]   --> /checkout/src/test/run-pass/realloc-16687.rs:82:9
[00:54:17]    |
[00:54:17] 70 |     unsafe fn reallocate(ptr: *mut u8, old: Layout, new: Layout) -> *mut u8 {
[00:54:17]    |                                                                     ------- expected `*mut u8` because of return type
[00:54:17] ...
[00:54:17] 82 |         ret.as_ptr()
[00:54:17]    |         ^^^^^^^^^^^^ expected u8, found extern type `std::alloc::Void`
[00:54:17]    |
[00:54:17]    = note: expected type `*mut u8`
[00:54:17]               found type `*mut std::alloc::Void`
---
[00:54:17] thread '[run-pass] run-pass/realloc-16687.rs' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:2901:9
---
[00:54:17] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/run-pass" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "run-pass" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-3.9/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Zmiri -Zunstable-options " "--target-rustcflags" "-Crpath -O -Zmiri -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "3.9.1\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[00:54:17] expected success, got: exit code: 101
[00:54:17]
[00:54:17]
[00:54:17] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[00:54:17] Build completed unsuccessfully in 0:13:47
[00:54:17] make: *** [check] Error 1
[00:54:17] Makefile:58: recipe for target 'check' failed

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN.

@Amanieu
Copy link
Member

Amanieu commented Apr 4, 2018

Unlike Alloc, it uses bare *mut Void without NonNull or Result.

I don't understand the reasoning behind this. Isn't *mut Void exactly equivalent to Result<NonNull<Void>, AllocErr>? There's even a From implementation to convert between them.

@sfackler
Copy link
Member

sfackler commented Apr 4, 2018

@Amanieu no one will be using the GlobalAlloc APIs directly, so we want to optimize towards ease of implementation rather than ease of consumption.

@SimonSapin
Copy link
Contributor Author

Well if it’s the only thing available on Stable, people (at least some library authors) will definitely use GlobalAlloc directly.

@SimonSapin
Copy link
Contributor Author

These tests already regressed at an intermediate commit for this PR and rust-lang/llvm#110 fixed them at the time, but now they regressed again and I don’t know why.

test [codegen] codegen/alloc-optimisation.rs ... FAILED
test [codegen] codegen/vec-optimizes-away.rs ... FAILED
failures:

---- [codegen] codegen/alloc-optimisation.rs stdout ----
	
error: verification with 'FileCheck' failed
status: exit code: 1
command: "/home/simon/rust1/build/x86_64-unknown-linux-gnu/llvm/build/bin/FileCheck" "--input-file" "/home/simon/rust1/build/x86_64-unknown-linux-gnu/test/codegen/alloc-optimisation.ll" "/home/simon/rust1/src/test/codegen/alloc-optimisation.rs"
stdout:
------------------------------------------

------------------------------------------
stderr:
------------------------------------------
/home/simon/rust1/src/test/codegen/alloc-optimisation.rs:19:17: error: CHECK-NEXT: is not on the line after the previous match
 // CHECK-NEXT: ret void
                ^
/home/simon/rust1/build/x86_64-unknown-linux-gnu/test/codegen/alloc-optimisation.ll:52:2: note: 'next' match was here
 ret void
 ^
/home/simon/rust1/build/x86_64-unknown-linux-gnu/test/codegen/alloc-optimisation.ll:40:7: note: previous match ended here
start:
      ^
/home/simon/rust1/build/x86_64-unknown-linux-gnu/test/codegen/alloc-optimisation.ll:41:1: note: non-matching line after previous match is here
 %0 = tail call i8* @__rust_alloc(i64 4, i64 4) #6
^

------------------------------------------

thread '[codegen] codegen/alloc-optimisation.rs' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:2901:9
note: Run with `RUST_BACKTRACE=1` for a backtrace.

---- [codegen] codegen/vec-optimizes-away.rs stdout ----
	
error: verification with 'FileCheck' failed
status: exit code: 1
command: "/home/simon/rust1/build/x86_64-unknown-linux-gnu/llvm/build/bin/FileCheck" "--input-file" "/home/simon/rust1/build/x86_64-unknown-linux-gnu/test/codegen/vec-optimizes-away.ll" "/home/simon/rust1/src/test/codegen/vec-optimizes-away.rs"
stdout:
------------------------------------------

------------------------------------------
stderr:
------------------------------------------
/home/simon/rust1/src/test/codegen/vec-optimizes-away.rs:19:17: error: CHECK-NEXT: is not on the line after the previous match
 // CHECK-NEXT: ret i32 6
                ^
/home/simon/rust1/build/x86_64-unknown-linux-gnu/test/codegen/vec-optimizes-away.ll:52:2: note: 'next' match was here
 ret i32 6
 ^
/home/simon/rust1/build/x86_64-unknown-linux-gnu/test/codegen/vec-optimizes-away.ll:40:7: note: previous match ended here
start:
      ^
/home/simon/rust1/build/x86_64-unknown-linux-gnu/test/codegen/vec-optimizes-away.ll:41:1: note: non-matching line after previous match is here
 %0 = tail call i8* @__rust_alloc(i64 12, i64 4) #6
^

------------------------------------------

thread '[codegen] codegen/vec-optimizes-away.rs' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:2901:9

@sfackler
Copy link
Member

sfackler commented Apr 4, 2018

What is the allocator going to be doing in the oom method? I don't really see why e.g. jemalloc would handle that differently than the system allocator.

@shepmaster
Copy link
Member

A little too big for me to handle 😉

r? @alexcrichton

@Amanieu
Copy link
Member

Amanieu commented Apr 5, 2018

You are right that jemalloc would handle this the same way as the system allocator. However in a no_std environment you need to be able to customize this behavior, for example to support customized logging (to a serial port, without allocating memory) or to reset the system to a clean state to avoid downtime.

There are two ways to support this functionality:

  • using a dynamic global OOM handler in liballoc which can changed with set_oom_handler.
  • adding an oom method to the GlobalAlloc trait.

Both of these will work for no_std systems so I will be happy with either of them. However I feel that the second option feels "cleaner" since it mirrors the oom function in the Alloc trait.

@sfackler
Copy link
Member

sfackler commented Apr 5, 2018

But if you want to customize that behavior then sticking it on GlobalAlloc is exactly what you wouldn't want, right? If I'm using dlmalloc in my kernel or whatever, I'm stuck with whatever oom implementation it decided on.

@TimNN
Copy link
Contributor

TimNN commented Apr 5, 2018

Your PR failed on Travis. Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
Receiving objects: 100% (338/338), 52.89 KiB | 17.63 MiB/s, done.
---
Resolving deltas: 100% (275/275), completed with 77 local objects.
---
[00:00:00] Attempting with retry: sh -c rm -f download-src-llvm.tar.gz &&         curl -sSL -o download-src-llvm.tar.gz https://github.com/rust-lang/llvm/archive/7243155b1c3da0a980c868a87adebf00e0b33989.tar.gz
---
[00:00:45] configure: rust.quiet-tests     := True
---
[00:43:12] ..........................................................................i.........................
[00:43:18] .................i..................................................................................
---
[00:43:55] ............................................................................................i.......
[00:44:02] ................................................................i...................................
---
[00:44:59] .............................................i......................................................
---
[00:49:06] .............................i......................................................................
[00:49:21] ..............................................................i.....................................
[00:49:37] ...............................................i....................................................
[00:49:58] ....................................................................................................
[00:50:21] ....................................................................................................
[00:50:44] ....................................................................................................
[00:51:11] .i...............................................................................................i..
[00:51:36] .......................................................................test [run-pass] run-pass/mir_heavy_promoted.rs has been running for over 60 seconds
[00:51:50] .............................
[00:52:21] ....................................................................................................
[00:53:00] .............................................................ii.....................................
[00:53:43] ........................i................................................test [run-pass] run-pass/saturating-float-casts.rs has been running for over 60 seconds
[00:53:54] ....i.ii...................
[00:54:39] .....................................................................................iiiiiii........
---
[00:56:38] ....................................i...............................................................
[00:56:46] ....................................................................................................
[00:56:54] ..................i............................................................ii.iii...............
[00:57:02] ....................................................................................................
[00:57:10] ........i..............................i............................................................
[00:57:18] ....................................................................................................
[00:57:25] ....................i...............................................................................
[00:57:34] ....................................................................................................
[00:57:44] ....................................................................................................
[00:57:55] ....................................................................................................
[00:58:06] ....................................................................................................
[00:58:20] ....................................................................................................
[00:58:29] .............i......................................................................................
[00:58:39] ................i..ii...............................................................................
[00:58:49] ....................................................................................................
[00:59:00] ....................................................................................................
[00:59:10] ..................................................................................i.................
[00:59:21] ............................i.......................................................................
---
[00:59:59] ............................i.......................................................................
[01:00:00] ....................................................................i...............................
[01:00:01] ................i.......................................................
---
[01:00:17] ...........i........................
---
[01:00:47] i...i..ii....i.............ii........iii......i..i...i...ii..i..i..ii.....
---
[01:00:50] i.......i......................i......
---
[01:01:31] iiii.......i..i........i..i.i.............i..........iiii...........i...i..........ii.i.i.......ii..
[01:01:32] ....ii...
---
[01:11:16] ...i................................................................................................
---
[01:13:15] .....................................i..............................................................
[01:13:36] ....................................................................................................
[01:13:59] ..........................................i.........................................................
---
[01:15:35] .........................................................ii.........................................
---
[01:16:45] ..............................................................i.....................................
---
[01:21:56] ii..................................................................................................
[01:22:16] ....................................................................................................
[01:22:33] ...................iii......i......i...i......i.....................................................
[01:22:43] ....................................................................................................
[01:23:00] ........................................iiii........ii..............................................
[01:23:12] ....................................................................................................
[01:23:31] .......................................................................................i............
---
[01:34:18] make[1]: Entering directory '/checkout/src/test/run-make-fulldeps/alloc-extern-crates'
[01:34:18] LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/alloc-extern-crates.stage2-x86_64-unknown-linux-gnu:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib:" '/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/alloc-extern-crates.stage2-x86_64-unknown-linux-gnu -L /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/alloc-extern-crates.stage2-x86_64-unknown-linux-gnu  fakealloc.rs
[01:34:18] LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/alloc-extern-crates.stage2-x86_64-unknown-linux-gnu:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib:" '/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/alloc-extern-crates.stage2-x86_64-unknown-linux-gnu -L /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/alloc-extern-crates.stage2-x86_64-unknown-linux-gnu  --crate-type=rlib ../../../liballoc/lib.rs --cfg feature=\"external_crate\" --extern external=/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/alloc-extern-crates.stage2-x86_64-unknown-linux-gnu/libfakealloc.rlib
[01:34:18] Makefile:4: recipe for target 'all' failed
[01:34:18] make[1]: Leaving directory '/checkout/src/test/run-make-fulldeps/alloc-extern-crates'
[01:34:18]
[01:34:18] ------------------------------------------
[01:34:18] stderr:
[01:34:18] ------------------------------------------
[01:34:18] error[E0464]: multiple matching crates for `libc`
[01:34:18]   --> ../../../liballoc/oom.rs:22:5
[01:34:18]    |
[01:34:18] 22 |     extern crate libc;
[01:34:18]    |     ^^^^^^^^^^^^^^^^^^
[01:34:18]    |
[01:34:18]    = note: candidates:
[01:34:18]            crate `libc`: /checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-b33d114295b26dd7.rlib
[01:34:18]            crate `libc`: /checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-b70301e459f49241.rlib
[01:34:18]
[01:34:18] error[E0463]: can't find crate for `libc`
[01:34:18]   --> ../../../liballoc/oom.rs:22:5
---
[01:34:18] make[1]: *** [all] Error 101
[01:34:18]
[01:34:18] ------------------------------------------
[01:34:18]
[01:34:18] thread '[run-make] run-make-fulldeps/alloc-extern-crates' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:2901:9
---
[01:34:18] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--rustdoc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "--src-base" "/checkout/src/test/run-make-fulldeps" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "run-make" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-3.9/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Zmiri -Zunstable-options " "--target-rustcflags" "-Crpath -O -Zmiri -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "3.9.1\n" "--system-llvm" "--cc" "cc" "--cxx" "c++" "--cflags" "-ffunction-sections -fdata-sections -fPIC -m64" "--llvm-components" "aarch64 aarch64asmparser aarch64asmprinter aarch64codegen aarch64desc aarch64disassembler aarch64info aarch64utils all all-targets amdgpu amdgpuasmparser amdgpuasmprinter amdgpucodegen amdgpudesc amdgpudisassembler amdgpuinfo amdgpuutils analysis arm armasmparser armasmprinter armcodegen armdesc armdisassembler arminfo asmparser asmprinter bitreader bitwriter bpf bpfasmprinter bpfcodegen bpfdesc bpfinfo codegen core coverage debuginfocodeview debuginfodwarf debuginfopdb engine executionengine globalisel hexagon hexagonasmparser hexagoncodegen hexagondesc hexagondisassembler hexagoninfo instcombine instrumentation interpreter ipo irreader libdriver lineeditor linker lto mc mcdisassembler mcjit mcparser mips mipsasmparser mipsasmprinter mipscodegen mipsdesc mipsdisassembler mipsinfo mirparser msp430 msp430asmprinter msp430codegen msp430desc msp430info native nativecodegen nvptx nvptxasmprinter nvptxcodegen nvptxdesc nvptxinfo objcarcopts object objectyaml option orcjit passes powerpc powerpcasmparser powerpcasmprinter powerpccodegen powerpcdesc powerpcdisassembler powerpcinfo profiledata runtimedyld scalaropts selectiondag sparc sparcasmparser sparcasmprinter sparccodegen sparcdesc sparcdisassembler sparcinfo support symbolize systemz systemzasmparser systemzasmprinter systemzcodegen systemzdesc systemzdisassembler systemzinfo tablegen target transformutils vectorize x86 x86asmparser x86asmprinter x86codegen x86desc x86disassembler x86info x86utils xcore xcoreasmprinter xcorecodegen xcoredesc xcoredisassembler xcoreinfo" "--llvm-cxxflags" "-I/usr/lib/llvm-3.9/include -std=c++0x -gsplit-dwarf -Wl,-fuse-ld=gold -fPIC -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment -Werror=date-time -std=c++11 -ffunction-sections -fdata-sections -O2 -g -DNDEBUG  -fno-exceptions -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS" "--ar" "ar" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[01:34:18] expected success, got: exit code: 101
[01:34:18]
[01:34:18]
[01:34:18] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:34:18] Build completed unsuccessfully in 0:52:25
[01:34:18] Makefile:58: recipe for target 'check' failed
[01:34:18] make: *** [check] Error 1
---
$ ls -lat $HOME/Library/Logs/DiagnosticReports/
ls: cannot access /home/travis/Library/Logs/DiagnosticReports/: No such file or directory
travis_time:end:16bb1ac8:start=1522891386195427412,finish=1522891386216171929,duration=20744517
travis_fold:end:after_failure.2
travis_fold:start:after_failure.3
travis_time:start:267211e8
$ find $HOME/Library/Logs/DiagnosticReports -type f -name '*.crash' -not -name '*.stage2-*.crash' -not -name 'com.apple.CoreSimulator.CoreSimulatorService-*.crash' -exec printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" {} \; -exec head -750 {} \; -exec echo travis_fold":"end:crashlog \; || true
find: `/home/travis/Library/Logs/DiagnosticReports': No such file or directory
travis_time:end:267211e8:start=1522891386223904836,finish=1522891386231533004,duration=7628168
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:24eca259
$ dmesg | grep -i kill
[   11.284348] init: failsafe main process (1096) killed by TERM signal

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN.

@Amanieu
Copy link
Member

Amanieu commented Apr 5, 2018

@sfackler You could use the same argument to say that the oom method doesn't belong on the Alloc trait. I doubt kernels would use the dlmalloc crate directly since it requires libc for syscalls, but even then there's nothing stopping you from creating a new GlobalAlloc which forwards to dlmalloc while overriding the oom method.

@sfackler
Copy link
Member

sfackler commented Apr 5, 2018

Yeah, Alloc::oom doesn't make much sense to me either.

@bors
Copy link
Contributor

bors commented Apr 13, 2018

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 99d4886 to master...

@bors bors merged commit c5ffdd7 into rust-lang:master Apr 13, 2018
@SimonSapin SimonSapin deleted the global-alloc branch April 13, 2018 14:28
SimonSapin added a commit to SimonSapin/jemallocator that referenced this pull request Apr 15, 2018
SimonSapin added a commit to SimonSapin/jemallocator that referenced this pull request Apr 15, 2018
bors-servo pushed a commit to servo/servo that referenced this pull request Apr 15, 2018
Update jemallocator, fix for nightly-2018-04-15

CC https://github.com/alexcrichton/jemallocator/pull/40, rust-lang/rust#49669

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/20640)
<!-- Reviewable:end -->
SimonSapin added a commit to servo/servo that referenced this pull request Apr 15, 2018
CC https://github.com/alexcrichton/jemallocator/pull/40,
rust-lang/rust#49669

The new version of jemallocator requires a more recent jemalloc
which doesn’t build on our current Android toolchain.

Rather than figuring that out, duplicate ~70 lines from jemallocator
and use the older jemalloc-sys directly.
SimonSapin added a commit to servo/servo that referenced this pull request Apr 15, 2018
CC https://github.com/alexcrichton/jemallocator/pull/40,
rust-lang/rust#49669

The new version of jemallocator requires a more recent jemalloc
which doesn’t build on our current Android toolchain.

Rather than figuring that out, duplicate ~70 lines from jemallocator
and use the older jemalloc-sys directly.
SimonSapin added a commit to servo/servo that referenced this pull request Apr 15, 2018
CC https://github.com/alexcrichton/jemallocator/pull/40,
rust-lang/rust#49669

The new version of jemallocator requires a more recent jemalloc
which doesn’t build on our current Android toolchain.

Rather than figuring that out, duplicate ~70 lines from jemallocator
and use the older jemalloc-sys directly.
SimonSapin added a commit to servo/servo that referenced this pull request Apr 15, 2018
CC https://github.com/alexcrichton/jemallocator/pull/40,
rust-lang/rust#49669

The new version of jemallocator requires a more recent jemalloc
which doesn’t build on our current Android toolchain.

Rather than figuring that out, duplicate ~70 lines from jemallocator
and use the older jemalloc-sys directly.
dwrensha added a commit to dwrensha/seer that referenced this pull request Apr 15, 2018
jonhoo added a commit to jonhoo/arccstr that referenced this pull request Apr 15, 2018
SimonSapin added a commit to servo/servo that referenced this pull request Apr 16, 2018
CC https://github.com/alexcrichton/jemallocator/pull/40,
rust-lang/rust#49669

The new version of jemallocator requires a more recent jemalloc
https://github.com/alexcrichton/jemallocator/pull/34
which doesn’t build on our current Android toolchain
jemalloc/jemalloc#1175.

To avoid blocking on figuring that out, duplicate ~70 lines
from jemallocator and use the older jemalloc-sys directly.
bors-servo pushed a commit to servo/servo that referenced this pull request Apr 16, 2018
Fork the jemallocator crate, fix for nightly-2018-04-15

CC https://github.com/alexcrichton/jemallocator/pull/40, rust-lang/rust#49669

The new version of jemallocator requires a more recent jemalloc https://github.com/alexcrichton/jemallocator/pull/34 which doesn’t build on our current Android toolchain jemalloc/jemalloc#1175.

To avoid blocking on figuring that out, duplicate ~70 lines from jemallocator and use the older jemalloc-sys directly.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/20641)
<!-- Reviewable:end -->
@Woyten Woyten mentioned this pull request Apr 18, 2018
1 task
@gnzlbg
Copy link
Contributor

gnzlbg commented May 16, 2018

So this removed the specializations of Alloc for jemalloc that used xallocx and nallocx.

Why? And where should these be re-added?

@SimonSapin
Copy link
Contributor Author

These still exist in https://crates.io/crates/jemallocator. The alloc_jemalloc however only (indirectly) provides impl GlobalAlloc for Global, and this trait does not have shrink_in_place or alloc_excess methods. Also that crate will likely be removed entirely when when make Global = System the default.

Moggers pushed a commit to Moggers/servo that referenced this pull request Jun 13, 2018
CC https://github.com/alexcrichton/jemallocator/pull/40,
rust-lang/rust#49669

The new version of jemallocator requires a more recent jemalloc
https://github.com/alexcrichton/jemallocator/pull/34
which doesn’t build on our current Android toolchain
jemalloc/jemalloc#1175.

To avoid blocking on figuring that out, duplicate ~70 lines
from jemallocator and use the older jemalloc-sys directly.
bestsoftwaretopappreviews25 added a commit to bestsoftwaretopappreviews25/arccstr that referenced this pull request Sep 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-allocators Area: Custom and system allocators S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.