Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update threadpool to use a circbuf instead of a deque #329

Closed
carllerche opened this issue Apr 30, 2018 · 4 comments
Closed

Update threadpool to use a circbuf instead of a deque #329

carllerche opened this issue Apr 30, 2018 · 4 comments

Comments

@carllerche
Copy link
Member

Currently, tokio-threadpool uses crossbeam-deque for the work queue. However, this is not the best option due to:

  • The workers pop from the same end as stealers.
  • The deque implementation uses fences, which throws off tsan*.

In this issue, it is recommended to use circbuf instead.

* I believe that this failure could have been caused by tsan being confused by a fence (or it is a bug).

/cc @stjepang, @jeehoonkang
@carllerche
Copy link
Member Author

For reference, I have also included the tsan error:

tsan output
running 1 test
==================
WARNING: ThreadSanitizer: data race (pid=37973)
  Atomic write of size 8 at 0x7b1c0002bf20 by thread T19:
    #0 __tsan_atomic64_compare_exchange_val /checkout/src/libcompiler_builtins/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc:851 (hammer-c57977986e577825+0x8cae9)
    #1 core::sync::atomic::atomic_compare_exchange::h6731fadb7a83d744 /checkout/src/libcore/sync/atomic.rs:1630 (hammer-c57977986e577825+0x109309)
    #2 core::sync::atomic::AtomicUsize::compare_exchange::h4ed1f07663d5ac77 /checkout/src/libcore/sync/atomic.rs:1229 (hammer-c57977986e577825+0x12996e)
    #3 core::sync::atomic::AtomicUsize::compare_and_swap::hc6ce47d08438e943 /checkout/src/libcore/sync/atomic.rs:1175 (hammer-c57977986e577825+0x129844)
    #4 tokio_threadpool::task::Task::schedule::h8550178a87f005f6 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:234 (hammer-c57977986e577825+0x1059f8)
    #5 _$LT$tokio_threadpool..notifier..Notifier$u20$as$u20$futures..task_impl..Notify$GT$::notify::hf9168aa9bb72d694 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/notifier.rs:35 (hammer-c57977986e577825+0x139148)
    #6 _$LT$futures..task_impl..std..ArcWrapped$LT$T$GT$$u20$as$u20$futures..task_impl..Notify$GT$::notify::h53806521c28ae67c /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/std/mod.rs:685 (hammer-c57977986e577825+0x12ff59)
    #7 futures::task_impl::NotifyHandle::notify::h7a64ed6ce5675673 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:621 (hammer-c57977986e577825+0x174ff4)
    #8 futures::task_impl::core::TaskUnpark::notify::h6ab272d73390caad /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/core.rs:65 (hammer-c57977986e577825+0x174696)
    #9 futures::task_impl::std::TaskUnpark::notify::h0cc8da198bb2c52d /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/std/mod.rs:196 (hammer-c57977986e577825+0x16df62)
    #10 futures::task_impl::Task::notify::h01ecd4739c6476f7 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:119 (hammer-c57977986e577825+0x174f40)
    #11 _$LT$futures..sync..oneshot..Inner$LT$T$GT$$GT$::drop_tx::h404301b268507ee0 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/sync/oneshot.rs:214 (hammer-c57977986e577825+0x30c8e)
    #12 _$LT$futures..sync..oneshot..Sender$LT$T$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::h2aed6e9897f57f10 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/sync/oneshot.rs:375 (hammer-c57977986e577825+0x31749)
    #13 core::ptr::drop_in_place::h087e2da3775ae4cb /checkout/src/libcore/ptr.rs:59 (hammer-c57977986e577825+0x1d188)
    #14 _$LT$futures..sync..oneshot..Sender$LT$T$GT$$GT$::send::hb69a9f33f88ffe52 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/sync/oneshot.rs:325 (hammer-c57977986e577825+0x30ff5)
    #15 hammer::hammer::_$u7b$$u7b$closure$u7d$$u7d$::h60c8c7266a9d7219 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:91 (hammer-c57977986e577825+0x45428)
    #16 _$LT$futures..future..and_then..AndThen$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h6038d89ab53bfcfa /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/future/and_then.rs:34 (hammer-c57977986e577825+0x2bd08)
    #17 _$LT$core..result..Result$LT$T$C$$u20$E$GT$$GT$::map::h85e18963e55c3c32 /checkout/src/libcore/result.rs:468 (hammer-c57977986e577825+0x191d0)
    #18 _$LT$futures..future..and_then..AndThen$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::h45ff77aa0c6fc277 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/future/and_then.rs:33 (hammer-c57977986e577825+0x2bc38)
    #19 _$LT$futures..future..chain..Chain$LT$A$C$$u20$B$C$$u20$C$GT$$GT$::poll::he63989ff783bbba3 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/future/chain.rs:39 (hammer-c57977986e577825+0x2b669)
    #20 _$LT$futures..future..and_then..AndThen$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hb4af8575aaf29fae /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/future/and_then.rs:32 (hammer-c57977986e577825+0x2bbd8)
    #21 _$LT$hammer..hammer..Counted$LT$T$GT$$u20$as$u20$futures..future..Future$GT$::poll::hdec25aac9fde2d91 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:30 (hammer-c57977986e577825+0x44c5c)
    #22 _$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hc8b313ab120e092b /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/future/mod.rs:113 (hammer-c57977986e577825+0x10e6ec)
    #23 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::poll_future_notify::_$u7b$$u7b$closure$u7d$$u7d$::h5f9cdcc02ea69798 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:289 (hammer-c57977986e577825+0x135388)
    #24 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::enter::_$u7b$$u7b$closure$u7d$$u7d$::h26a02a63cfad5f45 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:363 (hammer-c57977986e577825+0x1358db)
    #25 futures::task_impl::std::set::hf984587eeb93b3cc /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/std/mod.rs:78 (hammer-c57977986e577825+0x12fa42)
    #26 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::enter::ha149ce9f75c55bd4 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:363 (hammer-c57977986e577825+0x13583e)
    #27 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::poll_future_notify::h330eeec02baeeb92 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:289 (hammer-c57977986e577825+0x135266)
    #28 tokio_threadpool::task::TaskFuture::poll::h3ee729c6b70c209a /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:292 (hammer-c57977986e577825+0x105cac)
    #29 tokio_threadpool::task::Task::run::_$u7b$$u7b$closure$u7d$$u7d$::hb737ac194cc5977f /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:165 (hammer-c57977986e577825+0xff5ad)
    #30 core::ops::function::FnOnce::call_once::h187d6bb9aa1b58ff /checkout/src/libcore/ops/function.rs:223 (hammer-c57977986e577825+0xf9710)
    #31 _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h610e9aa008480940 /checkout/src/libstd/panic.rs:296 (hammer-c57977986e577825+0x1130cf)
    #32 std::panicking::try::do_call::hbb9f64052a3d1fa3 /checkout/src/libstd/panicking.rs:306 (hammer-c57977986e577825+0x14486b)
    #33 __rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:102 (hammer-c57977986e577825+0x19b1ee)
    #34 std::panic::catch_unwind::hf8936694dbbda59c /checkout/src/libstd/panic.rs:361 (hammer-c57977986e577825+0x11323c)
    #35 tokio_threadpool::task::Task::run::hc5dc9f09396ef157 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:151 (hammer-c57977986e577825+0x104e9c)
    #36 tokio_threadpool::worker::Worker::run_task2::heb3e1ea1d9db08ed /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:520 (hammer-c57977986e577825+0x1173d3)
    #37 tokio_threadpool::worker::Worker::run_task::h61de036a84873436 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:420 (hammer-c57977986e577825+0x116ba9)
    #38 tokio_threadpool::worker::Worker::try_run_owned_task::hb53b2ba14f485cdd /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:360 (hammer-c57977986e577825+0x115e9b)
    #39 tokio_threadpool::worker::Worker::try_run_task::hc8fd1c0e2aa76a06 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:289 (hammer-c57977986e577825+0x1154e8)
    #40 tokio_threadpool::worker::Worker::run::h42f2a13700e9f53e /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:235 (hammer-c57977986e577825+0x11529e)
    #41 tokio_threadpool::worker::Worker::do_run::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h96ae7e3abaf68f25 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:119 (hammer-c57977986e577825+0x147307)
    #42 tokio_executor::global::with_default::_$u7b$$u7b$closure$u7d$$u7d$::hb899c5ed9bc1970a /home/travis/build/tokio-rs/tokio/tokio-executor/src/global.rs:176 (hammer-c57977986e577825+0x13bdad)
    #43 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::ha5d64a1ab361f9ba /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b604)
    #44 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::haa2dd0dbde5265ab /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a94a)
    #45 tokio_executor::global::with_default::h9d1a7d8dbb9a0ee8 /home/travis/build/tokio-rs/tokio/tokio-executor/src/global.rs:150 (hammer-c57977986e577825+0x13bc0f)
    #46 tokio_threadpool::worker::Worker::do_run::_$u7b$$u7b$closure$u7d$$u7d$::h4065f71bfa34f9e7 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:115 (hammer-c57977986e577825+0x1474c8)
    #47 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::h5ad0b12fa0a89763 /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b3a3)
    #48 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h32301f38dfac8d20 /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a648)
    #49 tokio_threadpool::worker::Worker::do_run::h9986b07afd6e6570 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:106 (hammer-c57977986e577825+0x115097)
    #50 tokio_threadpool::pool::Pool::spawn_thread::_$u7b$$u7b$closure$u7d$$u7d$::h692f79d8ec032f46 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:398 (hammer-c57977986e577825+0x148eb2)
    #51 std::sys_common::backtrace::__rust_begin_short_backtrace::h01e0e0650fddd788 /checkout/src/libstd/sys_common/backtrace.rs:136 (hammer-c57977986e577825+0x149c1f)
    #52 std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h9d6ddc09ca694ab2 /checkout/src/libstd/thread/mod.rs:406 (hammer-c57977986e577825+0x13ca03)
    #53 _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::ha5fadba8ebb4d993 /checkout/src/libstd/panic.rs:296 (hammer-c57977986e577825+0x11313f)
    #54 std::panicking::try::do_call::h35832c6ed3d077f8 /checkout/src/libstd/panicking.rs:306 (hammer-c57977986e577825+0x1447c8)
    #55 __rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:102 (hammer-c57977986e577825+0x19b1ee)
    #56 std::panic::catch_unwind::h4f6590405ca70495 /checkout/src/libstd/panic.rs:361 (hammer-c57977986e577825+0x1131bc)
    #57 std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::h9086deeaab7af644 /checkout/src/libstd/thread/mod.rs:405 (hammer-c57977986e577825+0x13c7d0)
    #58 _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h1832befc0e5ed971 /checkout/src/liballoc/boxed.rs:783 (hammer-c57977986e577825+0x13ca77)
    #59 _$LT$alloc..boxed..Box$LT$alloc..boxed..FnBox$LT$A$C$$u20$Output$u3d$R$GT$$u20$$u2b$$u20$$u27$a$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::hd924cfcad138b2dd /checkout/src/liballoc/boxed.rs:793 (hammer-c57977986e577825+0x1906f7)
    #60 std::sys_common::thread::start_thread::h3a263f619eb7375b /checkout/src/libstd/sys_common/thread.rs:24 (hammer-c57977986e577825+0x1906f7)
  Previous write of size 8 at 0x7b1c0002bf20 by thread T18:
    #0 malloc /checkout/src/libcompiler_builtins/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:666 (hammer-c57977986e577825+0x5765f)
    #1 alloc_system::platform::_$LT$impl$u20$alloc..allocator..Alloc$u20$for$u20$$RF$$u27$a$u20$alloc_system..System$GT$::alloc::he830b5d50970fd13 /checkout/src/liballoc_system/lib.rs:135 (hammer-c57977986e577825+0x45575)
    #2 __rg_alloc /checkout/src/librustc_tsan/lib.rs:27 (hammer-c57977986e577825+0x45575)
    #3 alloc::heap::exchange_malloc::hb73c054e334b99b4 /checkout/src/liballoc/heap.rs:241 (hammer-c57977986e577825+0x10ffbb)
    #4 _$LT$alloc..arc..Arc$LT$T$GT$$GT$::new::h5acd3ee498ce821d /checkout/src/liballoc/arc.rs:284 (hammer-c57977986e577825+0x131508)
    #5 _$LT$$RF$$u27$a$u20$tokio_threadpool..sender..Sender$u20$as$u20$tokio_executor..Executor$GT$::spawn::h32b12e3d26f5095d /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:171 (hammer-c57977986e577825+0xf94d8)
    #6 tokio_threadpool::sender::Sender::spawn::h7aaacbfb8a3b58d9 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:87 (hammer-c57977986e577825+0x24ded)
    #7 hammer::hammer::_$u7b$$u7b$closure$u7d$$u7d$::h7bbc28dc3c172eb0 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:68 (hammer-c57977986e577825+0x4529b)
    #8 _$LT$futures..stream..for_each..ForEach$LT$S$C$$u20$F$C$$u20$U$GT$$u20$as$u20$futures..future..Future$GT$::poll::h6ac1d47ed7a3432b /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/stream/for_each.rs:46 (hammer-c57977986e577825+0x1b61f)
    #9 _$LT$hammer..hammer..Counted$LT$T$GT$$u20$as$u20$futures..future..Future$GT$::poll::h33067f13db245552 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:30 (hammer-c57977986e577825+0x44b9c)
    #10 _$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hc8b313ab120e092b /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/future/mod.rs:113 (hammer-c57977986e577825+0x10e6ec)
    #11 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::poll_future_notify::_$u7b$$u7b$closure$u7d$$u7d$::h5f9cdcc02ea69798 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:289 (hammer-c57977986e577825+0x135388)
    #12 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::enter::_$u7b$$u7b$closure$u7d$$u7d$::h26a02a63cfad5f45 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:363 (hammer-c57977986e577825+0x1358db)
    #13 futures::task_impl::std::set::hf984587eeb93b3cc /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/std/mod.rs:78 (hammer-c57977986e577825+0x12fa42)
    #14 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::enter::ha149ce9f75c55bd4 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:363 (hammer-c57977986e577825+0x13583e)
    #15 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::poll_future_notify::h330eeec02baeeb92 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:289 (hammer-c57977986e577825+0x135266)
    #16 tokio_threadpool::task::TaskFuture::poll::h3ee729c6b70c209a /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:292 (hammer-c57977986e577825+0x105cac)
    #17 tokio_threadpool::task::Task::run::_$u7b$$u7b$closure$u7d$$u7d$::hb737ac194cc5977f /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:165 (hammer-c57977986e577825+0xff5ad)
    #18 core::ops::function::FnOnce::call_once::h187d6bb9aa1b58ff /checkout/src/libcore/ops/function.rs:223 (hammer-c57977986e577825+0xf9710)
    #19 _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h610e9aa008480940 /checkout/src/libstd/panic.rs:296 (hammer-c57977986e577825+0x1130cf)
    #20 std::panicking::try::do_call::hbb9f64052a3d1fa3 /checkout/src/libstd/panicking.rs:306 (hammer-c57977986e577825+0x14486b)
    #21 __rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:102 (hammer-c57977986e577825+0x19b1ee)
    #22 std::panic::catch_unwind::hf8936694dbbda59c /checkout/src/libstd/panic.rs:361 (hammer-c57977986e577825+0x11323c)
    #23 tokio_threadpool::task::Task::run::hc5dc9f09396ef157 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:151 (hammer-c57977986e577825+0x104e9c)
    #24 tokio_threadpool::worker::Worker::run_task2::heb3e1ea1d9db08ed /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:520 (hammer-c57977986e577825+0x1173d3)
    #25 tokio_threadpool::worker::Worker::run_task::h61de036a84873436 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:420 (hammer-c57977986e577825+0x116ba9)
    #26 tokio_threadpool::worker::Worker::try_run_owned_task::hb53b2ba14f485cdd /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:360 (hammer-c57977986e577825+0x115e9b)
    #27 tokio_threadpool::worker::Worker::try_run_task::hc8fd1c0e2aa76a06 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:289 (hammer-c57977986e577825+0x1154e8)
    #28 tokio_threadpool::worker::Worker::run::h42f2a13700e9f53e /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:235 (hammer-c57977986e577825+0x11529e)
    #29 tokio_threadpool::worker::Worker::do_run::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h96ae7e3abaf68f25 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:119 (hammer-c57977986e577825+0x147307)
    #30 tokio_executor::global::with_default::_$u7b$$u7b$closure$u7d$$u7d$::hb899c5ed9bc1970a /home/travis/build/tokio-rs/tokio/tokio-executor/src/global.rs:176 (hammer-c57977986e577825+0x13bdad)
    #31 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::ha5d64a1ab361f9ba /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b604)
    #32 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::haa2dd0dbde5265ab /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a94a)
    #33 tokio_executor::global::with_default::h9d1a7d8dbb9a0ee8 /home/travis/build/tokio-rs/tokio/tokio-executor/src/global.rs:150 (hammer-c57977986e577825+0x13bc0f)
    #34 tokio_threadpool::worker::Worker::do_run::_$u7b$$u7b$closure$u7d$$u7d$::h4065f71bfa34f9e7 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:115 (hammer-c57977986e577825+0x1474c8)
    #35 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::h5ad0b12fa0a89763 /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b3a3)
    #36 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h32301f38dfac8d20 /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a648)
    #37 tokio_threadpool::worker::Worker::do_run::h9986b07afd6e6570 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:106 (hammer-c57977986e577825+0x115097)
    #38 tokio_threadpool::pool::Pool::spawn_thread::_$u7b$$u7b$closure$u7d$$u7d$::h692f79d8ec032f46 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:398 (hammer-c57977986e577825+0x148eb2)
    #39 std::sys_common::backtrace::__rust_begin_short_backtrace::h01e0e0650fddd788 /checkout/src/libstd/sys_common/backtrace.rs:136 (hammer-c57977986e577825+0x149c1f)
    #40 std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h9d6ddc09ca694ab2 /checkout/src/libstd/thread/mod.rs:406 (hammer-c57977986e577825+0x13ca03)
    #41 _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::ha5fadba8ebb4d993 /checkout/src/libstd/panic.rs:296 (hammer-c57977986e577825+0x11313f)
    #42 std::panicking::try::do_call::h35832c6ed3d077f8 /checkout/src/libstd/panicking.rs:306 (hammer-c57977986e577825+0x1447c8)
    #43 __rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:102 (hammer-c57977986e577825+0x19b1ee)
    #44 std::panic::catch_unwind::h4f6590405ca70495 /checkout/src/libstd/panic.rs:361 (hammer-c57977986e577825+0x1131bc)
    #45 std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::h9086deeaab7af644 /checkout/src/libstd/thread/mod.rs:405 (hammer-c57977986e577825+0x13c7d0)
    #46 _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h1832befc0e5ed971 /checkout/src/liballoc/boxed.rs:783 (hammer-c57977986e577825+0x13ca77)
    #47 _$LT$alloc..boxed..Box$LT$alloc..boxed..FnBox$LT$A$C$$u20$Output$u3d$R$GT$$u20$$u2b$$u20$$u27$a$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::hd924cfcad138b2dd /checkout/src/liballoc/boxed.rs:793 (hammer-c57977986e577825+0x1906f7)
    #48 std::sys_common::thread::start_thread::h3a263f619eb7375b /checkout/src/libstd/sys_common/thread.rs:24 (hammer-c57977986e577825+0x1906f7)
  Location is heap block of size 104 at 0x7b1c0002bf10 allocated by thread T18:
    #0 malloc /checkout/src/libcompiler_builtins/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:666 (hammer-c57977986e577825+0x5765f)
    #1 alloc_system::platform::_$LT$impl$u20$alloc..allocator..Alloc$u20$for$u20$$RF$$u27$a$u20$alloc_system..System$GT$::alloc::he830b5d50970fd13 /checkout/src/liballoc_system/lib.rs:135 (hammer-c57977986e577825+0x45575)
    #2 __rg_alloc /checkout/src/librustc_tsan/lib.rs:27 (hammer-c57977986e577825+0x45575)
    #3 alloc::heap::exchange_malloc::hb73c054e334b99b4 /checkout/src/liballoc/heap.rs:241 (hammer-c57977986e577825+0x10ffbb)
    #4 _$LT$alloc..arc..Arc$LT$T$GT$$GT$::new::h5acd3ee498ce821d /checkout/src/liballoc/arc.rs:284 (hammer-c57977986e577825+0x131508)
    #5 _$LT$$RF$$u27$a$u20$tokio_threadpool..sender..Sender$u20$as$u20$tokio_executor..Executor$GT$::spawn::h32b12e3d26f5095d /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:171 (hammer-c57977986e577825+0xf94d8)
    #6 tokio_threadpool::sender::Sender::spawn::h7aaacbfb8a3b58d9 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:87 (hammer-c57977986e577825+0x24ded)
    #7 hammer::hammer::_$u7b$$u7b$closure$u7d$$u7d$::h7bbc28dc3c172eb0 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:68 (hammer-c57977986e577825+0x4529b)
    #8 _$LT$futures..stream..for_each..ForEach$LT$S$C$$u20$F$C$$u20$U$GT$$u20$as$u20$futures..future..Future$GT$::poll::h6ac1d47ed7a3432b /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/stream/for_each.rs:46 (hammer-c57977986e577825+0x1b61f)
    #9 _$LT$hammer..hammer..Counted$LT$T$GT$$u20$as$u20$futures..future..Future$GT$::poll::h33067f13db245552 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:30 (hammer-c57977986e577825+0x44b9c)
    #10 _$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hc8b313ab120e092b /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/future/mod.rs:113 (hammer-c57977986e577825+0x10e6ec)
    #11 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::poll_future_notify::_$u7b$$u7b$closure$u7d$$u7d$::h5f9cdcc02ea69798 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:289 (hammer-c57977986e577825+0x135388)
    #12 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::enter::_$u7b$$u7b$closure$u7d$$u7d$::h26a02a63cfad5f45 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:363 (hammer-c57977986e577825+0x1358db)
    #13 futures::task_impl::std::set::hf984587eeb93b3cc /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/std/mod.rs:78 (hammer-c57977986e577825+0x12fa42)
    #14 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::enter::ha149ce9f75c55bd4 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:363 (hammer-c57977986e577825+0x13583e)
    #15 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::poll_future_notify::h330eeec02baeeb92 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:289 (hammer-c57977986e577825+0x135266)
    #16 tokio_threadpool::task::TaskFuture::poll::h3ee729c6b70c209a /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:292 (hammer-c57977986e577825+0x105cac)
    #17 tokio_threadpool::task::Task::run::_$u7b$$u7b$closure$u7d$$u7d$::hb737ac194cc5977f /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:165 (hammer-c57977986e577825+0xff5ad)
    #18 core::ops::function::FnOnce::call_once::h187d6bb9aa1b58ff /checkout/src/libcore/ops/function.rs:223 (hammer-c57977986e577825+0xf9710)
    #19 _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h610e9aa008480940 /checkout/src/libstd/panic.rs:296 (hammer-c57977986e577825+0x1130cf)
    #20 std::panicking::try::do_call::hbb9f64052a3d1fa3 /checkout/src/libstd/panicking.rs:306 (hammer-c57977986e577825+0x14486b)
    #21 __rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:102 (hammer-c57977986e577825+0x19b1ee)
    #22 std::panic::catch_unwind::hf8936694dbbda59c /checkout/src/libstd/panic.rs:361 (hammer-c57977986e577825+0x11323c)
    #23 tokio_threadpool::task::Task::run::hc5dc9f09396ef157 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:151 (hammer-c57977986e577825+0x104e9c)
    #24 tokio_threadpool::worker::Worker::run_task2::heb3e1ea1d9db08ed /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:520 (hammer-c57977986e577825+0x1173d3)
    #25 tokio_threadpool::worker::Worker::run_task::h61de036a84873436 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:420 (hammer-c57977986e577825+0x116ba9)
    #26 tokio_threadpool::worker::Worker::try_run_owned_task::hb53b2ba14f485cdd /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:360 (hammer-c57977986e577825+0x115e9b)
    #27 tokio_threadpool::worker::Worker::try_run_task::hc8fd1c0e2aa76a06 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:289 (hammer-c57977986e577825+0x1154e8)
    #28 tokio_threadpool::worker::Worker::run::h42f2a13700e9f53e /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:235 (hammer-c57977986e577825+0x11529e)
    #29 tokio_threadpool::worker::Worker::do_run::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h96ae7e3abaf68f25 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:119 (hammer-c57977986e577825+0x147307)
    #30 tokio_executor::global::with_default::_$u7b$$u7b$closure$u7d$$u7d$::hb899c5ed9bc1970a /home/travis/build/tokio-rs/tokio/tokio-executor/src/global.rs:176 (hammer-c57977986e577825+0x13bdad)
    #31 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::ha5d64a1ab361f9ba /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b604)
    #32 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::haa2dd0dbde5265ab /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a94a)
    #33 tokio_executor::global::with_default::h9d1a7d8dbb9a0ee8 /home/travis/build/tokio-rs/tokio/tokio-executor/src/global.rs:150 (hammer-c57977986e577825+0x13bc0f)
    #34 tokio_threadpool::worker::Worker::do_run::_$u7b$$u7b$closure$u7d$$u7d$::h4065f71bfa34f9e7 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:115 (hammer-c57977986e577825+0x1474c8)
    #35 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::h5ad0b12fa0a89763 /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b3a3)
    #36 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h32301f38dfac8d20 /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a648)
    #37 tokio_threadpool::worker::Worker::do_run::h9986b07afd6e6570 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:106 (hammer-c57977986e577825+0x115097)
    #38 tokio_threadpool::pool::Pool::spawn_thread::_$u7b$$u7b$closure$u7d$$u7d$::h692f79d8ec032f46 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:398 (hammer-c57977986e577825+0x148eb2)
    #39 std::sys_common::backtrace::__rust_begin_short_backtrace::h01e0e0650fddd788 /checkout/src/libstd/sys_common/backtrace.rs:136 (hammer-c57977986e577825+0x149c1f)
    #40 std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h9d6ddc09ca694ab2 /checkout/src/libstd/thread/mod.rs:406 (hammer-c57977986e577825+0x13ca03)
    #41 _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::ha5fadba8ebb4d993 /checkout/src/libstd/panic.rs:296 (hammer-c57977986e577825+0x11313f)
    #42 std::panicking::try::do_call::h35832c6ed3d077f8 /checkout/src/libstd/panicking.rs:306 (hammer-c57977986e577825+0x1447c8)
    #43 __rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:102 (hammer-c57977986e577825+0x19b1ee)
    #44 std::panic::catch_unwind::h4f6590405ca70495 /checkout/src/libstd/panic.rs:361 (hammer-c57977986e577825+0x1131bc)
    #45 std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::h9086deeaab7af644 /checkout/src/libstd/thread/mod.rs:405 (hammer-c57977986e577825+0x13c7d0)
    #46 _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h1832befc0e5ed971 /checkout/src/liballoc/boxed.rs:783 (hammer-c57977986e577825+0x13ca77)
    #47 _$LT$alloc..boxed..Box$LT$alloc..boxed..FnBox$LT$A$C$$u20$Output$u3d$R$GT$$u20$$u2b$$u20$$u27$a$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::hd924cfcad138b2dd /checkout/src/liballoc/boxed.rs:793 (hammer-c57977986e577825+0x1906f7)
    #48 std::sys_common::thread::start_thread::h3a263f619eb7375b /checkout/src/libstd/sys_common/thread.rs:24 (hammer-c57977986e577825+0x1906f7)
  Thread T19 (tid=37994, running) created by thread T1 at:
    #0 pthread_create /checkout/src/libcompiler_builtins/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (hammer-c57977986e577825+0x4d2a6)
    #1 std::sys::unix::thread::Thread::new::he5ce8706496dfcd2 /checkout/src/libstd/sys/unix/thread.rs:78 (hammer-c57977986e577825+0x1832f9)
    #2 tokio_threadpool::pool::Pool::spawn_thread::hff5239497e631c9c /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:380 (hammer-c57977986e577825+0x11e368)
    #3 tokio_threadpool::pool::Pool::submit_to_external::h6944433290a53d91 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:328 (hammer-c57977986e577825+0x11d6e4)
    #4 tokio_threadpool::pool::Pool::submit_external::h3783cdd23519a2e7 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:304 (hammer-c57977986e577825+0x11cfef)
    #5 tokio_threadpool::pool::Pool::submit::_$u7b$$u7b$closure$u7d$$u7d$::h830d8c681255f053 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:287 (hammer-c57977986e577825+0x148617)
    #6 tokio_threadpool::worker::Worker::with_current::_$u7b$$u7b$closure$u7d$$u7d$::hac6f7b867f2cb3a0 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:135 (hammer-c57977986e577825+0x1476bb)
    #7 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::h384dc84a000e44c1 /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b109)
    #8 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h962936840c1528c9 /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a89a)
    #9 tokio_threadpool::worker::Worker::with_current::h5bf3067078fbcd34 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:131 (hammer-c57977986e577825+0x1475b9)
    #10 tokio_threadpool::pool::Pool::submit::hd1cd6a9d64c5c7ea /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:272 (hammer-c57977986e577825+0x11c9f8)
    #11 _$LT$$RF$$u27$a$u20$tokio_threadpool..sender..Sender$u20$as$u20$tokio_executor..Executor$GT$::spawn::h32b12e3d26f5095d /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:173 (hammer-c57977986e577825+0xf956f)
    #12 tokio_threadpool::sender::Sender::spawn::he5446b8adab58cef /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:87 (hammer-c57977986e577825+0x24f74)
    #13 tokio_threadpool::thread_pool::ThreadPool::spawn::hb646213754578c25 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/thread_pool.rs:64 (hammer-c57977986e577825+0x43ca7)
    #14 hammer::hammer::hc9f00956ee3ac9e9 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:87 (hammer-c57977986e577825+0x29006)
    #15 hammer::__test::TESTS::_$u7b$$u7b$closure$u7d$$u7d$::h21abcb5b42c22027 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:13 (hammer-c57977986e577825+0x2bb04)
    #16 core::ops::function::FnOnce::call_once::h7bb843658796ab47 /checkout/src/libcore/ops/function.rs:223 (hammer-c57977986e577825+0x1cc29)
    #17 test::run_test::_$u7b$$u7b$closure$u7d$$u7d$::h648e6b00b9b34d4c /checkout/src/libtest/lib.rs:1453 (hammer-c57977986e577825+0xbb0b1)
    #18 core::ops::function::FnOnce::call_once::hab331e63f4b9d590 /checkout/src/libcore/ops/function.rs:223 (hammer-c57977986e577825+0xbb0b1)
    #19 _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h8c9806229337e6a4 /checkout/src/liballoc/boxed.rs:783 (hammer-c57977986e577825+0xbb0b1)
  Thread T18 (tid=37993, running) created by thread T1 at:
    #0 pthread_create /checkout/src/libcompiler_builtins/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (hammer-c57977986e577825+0x4d2a6)
    #1 std::sys::unix::thread::Thread::new::he5ce8706496dfcd2 /checkout/src/libstd/sys/unix/thread.rs:78 (hammer-c57977986e577825+0x1832f9)
    #2 tokio_threadpool::pool::Pool::spawn_thread::hff5239497e631c9c /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:380 (hammer-c57977986e577825+0x11e368)
    #3 tokio_threadpool::pool::Pool::submit_to_external::h6944433290a53d91 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:328 (hammer-c57977986e577825+0x11d6e4)
    #4 tokio_threadpool::pool::Pool::submit_external::h3783cdd23519a2e7 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:304 (hammer-c57977986e577825+0x11cfef)
    #5 tokio_threadpool::pool::Pool::submit::_$u7b$$u7b$closure$u7d$$u7d$::h830d8c681255f053 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:287 (hammer-c57977986e577825+0x148617)
    #6 tokio_threadpool::worker::Worker::with_current::_$u7b$$u7b$closure$u7d$$u7d$::hac6f7b867f2cb3a0 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:135 (hammer-c57977986e577825+0x1476bb)
    #7 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::h384dc84a000e44c1 /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b109)
    #8 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h962936840c1528c9 /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a89a)
    #9 tokio_threadpool::worker::Worker::with_current::h5bf3067078fbcd34 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:131 (hammer-c57977986e577825+0x1475b9)
    #10 tokio_threadpool::pool::Pool::submit::hd1cd6a9d64c5c7ea /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:272 (hammer-c57977986e577825+0x11c9f8)
    #11 _$LT$$RF$$u27$a$u20$tokio_threadpool..sender..Sender$u20$as$u20$tokio_executor..Executor$GT$::spawn::h32b12e3d26f5095d /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:173 (hammer-c57977986e577825+0xf956f)
    #12 tokio_threadpool::sender::Sender::spawn::h4d3ab4e394b134c0 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:87 (hammer-c57977986e577825+0x24c5d)
    #13 tokio_threadpool::thread_pool::ThreadPool::spawn::h9596a22fb8d14e46 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/thread_pool.rs:64 (hammer-c57977986e577825+0x43b97)
    #14 hammer::hammer::hc9f00956ee3ac9e9 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:50 (hammer-c57977986e577825+0x28d3f)
    #15 hammer::__test::TESTS::_$u7b$$u7b$closure$u7d$$u7d$::h21abcb5b42c22027 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:13 (hammer-c57977986e577825+0x2bb04)
    #16 core::ops::function::FnOnce::call_once::h7bb843658796ab47 /checkout/src/libcore/ops/function.rs:223 (hammer-c57977986e577825+0x1cc29)
    #17 test::run_test::_$u7b$$u7b$closure$u7d$$u7d$::h648e6b00b9b34d4c /checkout/src/libtest/lib.rs:1453 (hammer-c57977986e577825+0xbb0b1)
    #18 core::ops::function::FnOnce::call_once::hab331e63f4b9d590 /checkout/src/libcore/ops/function.rs:223 (hammer-c57977986e577825+0xbb0b1)
    #19 _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h8c9806229337e6a4 /checkout/src/liballoc/boxed.rs:783 (hammer-c57977986e577825+0xbb0b1)
SUMMARY: ThreadSanitizer: data race /checkout/src/libcore/sync/atomic.rs:1630 in core::sync::atomic::atomic_compare_exchange::h6731fadb7a83d744
==================
==================
WARNING: ThreadSanitizer: data race (pid=37973)
  Atomic write of size 8 at 0x7b1c0002bf10 by thread T19:
    #0 __tsan_atomic64_fetch_add /checkout/src/libcompiler_builtins/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc:621 (hammer-c57977986e577825+0x8bcd1)
    #1 core::sync::atomic::atomic_add::h39d213ff06ee7936 /checkout/src/libcore/sync/atomic.rs:1601 (hammer-c57977986e577825+0x10839f)
    #2 core::sync::atomic::AtomicUsize::fetch_add::h2719cc0280369ba2 /checkout/src/libcore/sync/atomic.rs:1299 (hammer-c57977986e577825+0x129d48)
    #3 _$LT$alloc..arc..Arc$LT$T$GT$$u20$as$u20$core..clone..Clone$GT$::clone::h729d3d656be2315a /checkout/src/liballoc/arc.rs:713 (hammer-c57977986e577825+0x133f88)
    #4 _$LT$tokio_threadpool..notifier..Notifier$u20$as$u20$futures..task_impl..Notify$GT$::notify::hf9168aa9bb72d694 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/notifier.rs:39 (hammer-c57977986e577825+0x13917d)
    #5 _$LT$futures..task_impl..std..ArcWrapped$LT$T$GT$$u20$as$u20$futures..task_impl..Notify$GT$::notify::h53806521c28ae67c /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/std/mod.rs:685 (hammer-c57977986e577825+0x12ff59)
    #6 futures::task_impl::NotifyHandle::notify::h7a64ed6ce5675673 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:621 (hammer-c57977986e577825+0x174ff4)
    #7 futures::task_impl::core::TaskUnpark::notify::h6ab272d73390caad /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/core.rs:65 (hammer-c57977986e577825+0x174696)
    #8 futures::task_impl::std::TaskUnpark::notify::h0cc8da198bb2c52d /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/std/mod.rs:196 (hammer-c57977986e577825+0x16df62)
    #9 futures::task_impl::Task::notify::h01ecd4739c6476f7 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:119 (hammer-c57977986e577825+0x174f40)
    #10 _$LT$futures..sync..oneshot..Inner$LT$T$GT$$GT$::drop_tx::h404301b268507ee0 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/sync/oneshot.rs:214 (hammer-c57977986e577825+0x30c8e)
    #11 _$LT$futures..sync..oneshot..Sender$LT$T$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::h2aed6e9897f57f10 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/sync/oneshot.rs:375 (hammer-c57977986e577825+0x31749)
    #12 core::ptr::drop_in_place::h087e2da3775ae4cb /checkout/src/libcore/ptr.rs:59 (hammer-c57977986e577825+0x1d188)
    #13 _$LT$futures..sync..oneshot..Sender$LT$T$GT$$GT$::send::hb69a9f33f88ffe52 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/sync/oneshot.rs:325 (hammer-c57977986e577825+0x30ff5)
    #14 hammer::hammer::_$u7b$$u7b$closure$u7d$$u7d$::h60c8c7266a9d7219 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:91 (hammer-c57977986e577825+0x45428)
    #15 _$LT$futures..future..and_then..AndThen$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h6038d89ab53bfcfa /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/future/and_then.rs:34 (hammer-c57977986e577825+0x2bd08)
    #16 _$LT$core..result..Result$LT$T$C$$u20$E$GT$$GT$::map::h85e18963e55c3c32 /checkout/src/libcore/result.rs:468 (hammer-c57977986e577825+0x191d0)
    #17 _$LT$futures..future..and_then..AndThen$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::h45ff77aa0c6fc277 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/future/and_then.rs:33 (hammer-c57977986e577825+0x2bc38)
    #18 _$LT$futures..future..chain..Chain$LT$A$C$$u20$B$C$$u20$C$GT$$GT$::poll::he63989ff783bbba3 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/future/chain.rs:39 (hammer-c57977986e577825+0x2b669)
    #19 _$LT$futures..future..and_then..AndThen$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hb4af8575aaf29fae /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/future/and_then.rs:32 (hammer-c57977986e577825+0x2bbd8)
    #20 _$LT$hammer..hammer..Counted$LT$T$GT$$u20$as$u20$futures..future..Future$GT$::poll::hdec25aac9fde2d91 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:30 (hammer-c57977986e577825+0x44c5c)
    #21 _$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hc8b313ab120e092b /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/future/mod.rs:113 (hammer-c57977986e577825+0x10e6ec)
    #22 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::poll_future_notify::_$u7b$$u7b$closure$u7d$$u7d$::h5f9cdcc02ea69798 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:289 (hammer-c57977986e577825+0x135388)
    #23 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::enter::_$u7b$$u7b$closure$u7d$$u7d$::h26a02a63cfad5f45 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:363 (hammer-c57977986e577825+0x1358db)
    #24 futures::task_impl::std::set::hf984587eeb93b3cc /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/std/mod.rs:78 (hammer-c57977986e577825+0x12fa42)
    #25 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::enter::ha149ce9f75c55bd4 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:363 (hammer-c57977986e577825+0x13583e)
    #26 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::poll_future_notify::h330eeec02baeeb92 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:289 (hammer-c57977986e577825+0x135266)
    #27 tokio_threadpool::task::TaskFuture::poll::h3ee729c6b70c209a /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:292 (hammer-c57977986e577825+0x105cac)
    #28 tokio_threadpool::task::Task::run::_$u7b$$u7b$closure$u7d$$u7d$::hb737ac194cc5977f /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:165 (hammer-c57977986e577825+0xff5ad)
    #29 core::ops::function::FnOnce::call_once::h187d6bb9aa1b58ff /checkout/src/libcore/ops/function.rs:223 (hammer-c57977986e577825+0xf9710)
    #30 _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h610e9aa008480940 /checkout/src/libstd/panic.rs:296 (hammer-c57977986e577825+0x1130cf)
    #31 std::panicking::try::do_call::hbb9f64052a3d1fa3 /checkout/src/libstd/panicking.rs:306 (hammer-c57977986e577825+0x14486b)
    #32 __rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:102 (hammer-c57977986e577825+0x19b1ee)
    #33 std::panic::catch_unwind::hf8936694dbbda59c /checkout/src/libstd/panic.rs:361 (hammer-c57977986e577825+0x11323c)
    #34 tokio_threadpool::task::Task::run::hc5dc9f09396ef157 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:151 (hammer-c57977986e577825+0x104e9c)
    #35 tokio_threadpool::worker::Worker::run_task2::heb3e1ea1d9db08ed /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:520 (hammer-c57977986e577825+0x1173d3)
    #36 tokio_threadpool::worker::Worker::run_task::h61de036a84873436 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:420 (hammer-c57977986e577825+0x116ba9)
    #37 tokio_threadpool::worker::Worker::try_run_owned_task::hb53b2ba14f485cdd /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:360 (hammer-c57977986e577825+0x115e9b)
    #38 tokio_threadpool::worker::Worker::try_run_task::hc8fd1c0e2aa76a06 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:289 (hammer-c57977986e577825+0x1154e8)
    #39 tokio_threadpool::worker::Worker::run::h42f2a13700e9f53e /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:235 (hammer-c57977986e577825+0x11529e)
    #40 tokio_threadpool::worker::Worker::do_run::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h96ae7e3abaf68f25 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:119 (hammer-c57977986e577825+0x147307)
    #41 tokio_executor::global::with_default::_$u7b$$u7b$closure$u7d$$u7d$::hb899c5ed9bc1970a /home/travis/build/tokio-rs/tokio/tokio-executor/src/global.rs:176 (hammer-c57977986e577825+0x13bdad)
    #42 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::ha5d64a1ab361f9ba /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b604)
    #43 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::haa2dd0dbde5265ab /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a94a)
    #44 tokio_executor::global::with_default::h9d1a7d8dbb9a0ee8 /home/travis/build/tokio-rs/tokio/tokio-executor/src/global.rs:150 (hammer-c57977986e577825+0x13bc0f)
    #45 tokio_threadpool::worker::Worker::do_run::_$u7b$$u7b$closure$u7d$$u7d$::h4065f71bfa34f9e7 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:115 (hammer-c57977986e577825+0x1474c8)
    #46 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::h5ad0b12fa0a89763 /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b3a3)
    #47 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h32301f38dfac8d20 /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a648)
    #48 tokio_threadpool::worker::Worker::do_run::h9986b07afd6e6570 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:106 (hammer-c57977986e577825+0x115097)
    #49 tokio_threadpool::pool::Pool::spawn_thread::_$u7b$$u7b$closure$u7d$$u7d$::h692f79d8ec032f46 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:398 (hammer-c57977986e577825+0x148eb2)
    #50 std::sys_common::backtrace::__rust_begin_short_backtrace::h01e0e0650fddd788 /checkout/src/libstd/sys_common/backtrace.rs:136 (hammer-c57977986e577825+0x149c1f)
    #51 std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h9d6ddc09ca694ab2 /checkout/src/libstd/thread/mod.rs:406 (hammer-c57977986e577825+0x13ca03)
    #52 _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::ha5fadba8ebb4d993 /checkout/src/libstd/panic.rs:296 (hammer-c57977986e577825+0x11313f)
    #53 std::panicking::try::do_call::h35832c6ed3d077f8 /checkout/src/libstd/panicking.rs:306 (hammer-c57977986e577825+0x1447c8)
    #54 __rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:102 (hammer-c57977986e577825+0x19b1ee)
    #55 std::panic::catch_unwind::h4f6590405ca70495 /checkout/src/libstd/panic.rs:361 (hammer-c57977986e577825+0x1131bc)
    #56 std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::h9086deeaab7af644 /checkout/src/libstd/thread/mod.rs:405 (hammer-c57977986e577825+0x13c7d0)
    #57 _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h1832befc0e5ed971 /checkout/src/liballoc/boxed.rs:783 (hammer-c57977986e577825+0x13ca77)
    #58 _$LT$alloc..boxed..Box$LT$alloc..boxed..FnBox$LT$A$C$$u20$Output$u3d$R$GT$$u20$$u2b$$u20$$u27$a$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::hd924cfcad138b2dd /checkout/src/liballoc/boxed.rs:793 (hammer-c57977986e577825+0x1906f7)
    #59 std::sys_common::thread::start_thread::h3a263f619eb7375b /checkout/src/libstd/sys_common/thread.rs:24 (hammer-c57977986e577825+0x1906f7)
  Previous write of size 8 at 0x7b1c0002bf10 by thread T18:
    #0 malloc /checkout/src/libcompiler_builtins/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:666 (hammer-c57977986e577825+0x5765f)
    #1 alloc_system::platform::_$LT$impl$u20$alloc..allocator..Alloc$u20$for$u20$$RF$$u27$a$u20$alloc_system..System$GT$::alloc::he830b5d50970fd13 /checkout/src/liballoc_system/lib.rs:135 (hammer-c57977986e577825+0x45575)
    #2 __rg_alloc /checkout/src/librustc_tsan/lib.rs:27 (hammer-c57977986e577825+0x45575)
    #3 alloc::heap::exchange_malloc::hb73c054e334b99b4 /checkout/src/liballoc/heap.rs:241 (hammer-c57977986e577825+0x10ffbb)
    #4 _$LT$alloc..arc..Arc$LT$T$GT$$GT$::new::h5acd3ee498ce821d /checkout/src/liballoc/arc.rs:284 (hammer-c57977986e577825+0x131508)
    #5 _$LT$$RF$$u27$a$u20$tokio_threadpool..sender..Sender$u20$as$u20$tokio_executor..Executor$GT$::spawn::h32b12e3d26f5095d /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:171 (hammer-c57977986e577825+0xf94d8)
    #6 tokio_threadpool::sender::Sender::spawn::h7aaacbfb8a3b58d9 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:87 (hammer-c57977986e577825+0x24ded)
    #7 hammer::hammer::_$u7b$$u7b$closure$u7d$$u7d$::h7bbc28dc3c172eb0 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:68 (hammer-c57977986e577825+0x4529b)
    #8 _$LT$futures..stream..for_each..ForEach$LT$S$C$$u20$F$C$$u20$U$GT$$u20$as$u20$futures..future..Future$GT$::poll::h6ac1d47ed7a3432b /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/stream/for_each.rs:46 (hammer-c57977986e577825+0x1b61f)
    #9 _$LT$hammer..hammer..Counted$LT$T$GT$$u20$as$u20$futures..future..Future$GT$::poll::h33067f13db245552 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:30 (hammer-c57977986e577825+0x44b9c)
    #10 _$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hc8b313ab120e092b /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/future/mod.rs:113 (hammer-c57977986e577825+0x10e6ec)
    #11 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::poll_future_notify::_$u7b$$u7b$closure$u7d$$u7d$::h5f9cdcc02ea69798 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:289 (hammer-c57977986e577825+0x135388)
    #12 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::enter::_$u7b$$u7b$closure$u7d$$u7d$::h26a02a63cfad5f45 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:363 (hammer-c57977986e577825+0x1358db)
    #13 futures::task_impl::std::set::hf984587eeb93b3cc /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/std/mod.rs:78 (hammer-c57977986e577825+0x12fa42)
    #14 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::enter::ha149ce9f75c55bd4 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:363 (hammer-c57977986e577825+0x13583e)
    #15 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::poll_future_notify::h330eeec02baeeb92 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:289 (hammer-c57977986e577825+0x135266)
    #16 tokio_threadpool::task::TaskFuture::poll::h3ee729c6b70c209a /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:292 (hammer-c57977986e577825+0x105cac)
    #17 tokio_threadpool::task::Task::run::_$u7b$$u7b$closure$u7d$$u7d$::hb737ac194cc5977f /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:165 (hammer-c57977986e577825+0xff5ad)
    #18 core::ops::function::FnOnce::call_once::h187d6bb9aa1b58ff /checkout/src/libcore/ops/function.rs:223 (hammer-c57977986e577825+0xf9710)
    #19 _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h610e9aa008480940 /checkout/src/libstd/panic.rs:296 (hammer-c57977986e577825+0x1130cf)
    #20 std::panicking::try::do_call::hbb9f64052a3d1fa3 /checkout/src/libstd/panicking.rs:306 (hammer-c57977986e577825+0x14486b)
    #21 __rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:102 (hammer-c57977986e577825+0x19b1ee)
    #22 std::panic::catch_unwind::hf8936694dbbda59c /checkout/src/libstd/panic.rs:361 (hammer-c57977986e577825+0x11323c)
    #23 tokio_threadpool::task::Task::run::hc5dc9f09396ef157 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:151 (hammer-c57977986e577825+0x104e9c)
    #24 tokio_threadpool::worker::Worker::run_task2::heb3e1ea1d9db08ed /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:520 (hammer-c57977986e577825+0x1173d3)
    #25 tokio_threadpool::worker::Worker::run_task::h61de036a84873436 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:420 (hammer-c57977986e577825+0x116ba9)
    #26 tokio_threadpool::worker::Worker::try_run_owned_task::hb53b2ba14f485cdd /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:360 (hammer-c57977986e577825+0x115e9b)
    #27 tokio_threadpool::worker::Worker::try_run_task::hc8fd1c0e2aa76a06 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:289 (hammer-c57977986e577825+0x1154e8)
    #28 tokio_threadpool::worker::Worker::run::h42f2a13700e9f53e /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:235 (hammer-c57977986e577825+0x11529e)
    #29 tokio_threadpool::worker::Worker::do_run::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h96ae7e3abaf68f25 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:119 (hammer-c57977986e577825+0x147307)
    #30 tokio_executor::global::with_default::_$u7b$$u7b$closure$u7d$$u7d$::hb899c5ed9bc1970a /home/travis/build/tokio-rs/tokio/tokio-executor/src/global.rs:176 (hammer-c57977986e577825+0x13bdad)
    #31 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::ha5d64a1ab361f9ba /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b604)
    #32 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::haa2dd0dbde5265ab /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a94a)
    #33 tokio_executor::global::with_default::h9d1a7d8dbb9a0ee8 /home/travis/build/tokio-rs/tokio/tokio-executor/src/global.rs:150 (hammer-c57977986e577825+0x13bc0f)
    #34 tokio_threadpool::worker::Worker::do_run::_$u7b$$u7b$closure$u7d$$u7d$::h4065f71bfa34f9e7 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:115 (hammer-c57977986e577825+0x1474c8)
    #35 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::h5ad0b12fa0a89763 /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b3a3)
    #36 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h32301f38dfac8d20 /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a648)
    #37 tokio_threadpool::worker::Worker::do_run::h9986b07afd6e6570 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:106 (hammer-c57977986e577825+0x115097)
    #38 tokio_threadpool::pool::Pool::spawn_thread::_$u7b$$u7b$closure$u7d$$u7d$::h692f79d8ec032f46 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:398 (hammer-c57977986e577825+0x148eb2)
    #39 std::sys_common::backtrace::__rust_begin_short_backtrace::h01e0e0650fddd788 /checkout/src/libstd/sys_common/backtrace.rs:136 (hammer-c57977986e577825+0x149c1f)
    #40 std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h9d6ddc09ca694ab2 /checkout/src/libstd/thread/mod.rs:406 (hammer-c57977986e577825+0x13ca03)
    #41 _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::ha5fadba8ebb4d993 /checkout/src/libstd/panic.rs:296 (hammer-c57977986e577825+0x11313f)
    #42 std::panicking::try::do_call::h35832c6ed3d077f8 /checkout/src/libstd/panicking.rs:306 (hammer-c57977986e577825+0x1447c8)
    #43 __rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:102 (hammer-c57977986e577825+0x19b1ee)
    #44 std::panic::catch_unwind::h4f6590405ca70495 /checkout/src/libstd/panic.rs:361 (hammer-c57977986e577825+0x1131bc)
    #45 std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::h9086deeaab7af644 /checkout/src/libstd/thread/mod.rs:405 (hammer-c57977986e577825+0x13c7d0)
    #46 _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h1832befc0e5ed971 /checkout/src/liballoc/boxed.rs:783 (hammer-c57977986e577825+0x13ca77)
    #47 _$LT$alloc..boxed..Box$LT$alloc..boxed..FnBox$LT$A$C$$u20$Output$u3d$R$GT$$u20$$u2b$$u20$$u27$a$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::hd924cfcad138b2dd /checkout/src/liballoc/boxed.rs:793 (hammer-c57977986e577825+0x1906f7)
    #48 std::sys_common::thread::start_thread::h3a263f619eb7375b /checkout/src/libstd/sys_common/thread.rs:24 (hammer-c57977986e577825+0x1906f7)
  Location is heap block of size 104 at 0x7b1c0002bf10 allocated by thread T18:
    #0 malloc /checkout/src/libcompiler_builtins/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:666 (hammer-c57977986e577825+0x5765f)
    #1 alloc_system::platform::_$LT$impl$u20$alloc..allocator..Alloc$u20$for$u20$$RF$$u27$a$u20$alloc_system..System$GT$::alloc::he830b5d50970fd13 /checkout/src/liballoc_system/lib.rs:135 (hammer-c57977986e577825+0x45575)
    #2 __rg_alloc /checkout/src/librustc_tsan/lib.rs:27 (hammer-c57977986e577825+0x45575)
    #3 alloc::heap::exchange_malloc::hb73c054e334b99b4 /checkout/src/liballoc/heap.rs:241 (hammer-c57977986e577825+0x10ffbb)
    #4 _$LT$alloc..arc..Arc$LT$T$GT$$GT$::new::h5acd3ee498ce821d /checkout/src/liballoc/arc.rs:284 (hammer-c57977986e577825+0x131508)
    #5 _$LT$$RF$$u27$a$u20$tokio_threadpool..sender..Sender$u20$as$u20$tokio_executor..Executor$GT$::spawn::h32b12e3d26f5095d /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:171 (hammer-c57977986e577825+0xf94d8)
    #6 tokio_threadpool::sender::Sender::spawn::h7aaacbfb8a3b58d9 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:87 (hammer-c57977986e577825+0x24ded)
    #7 hammer::hammer::_$u7b$$u7b$closure$u7d$$u7d$::h7bbc28dc3c172eb0 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:68 (hammer-c57977986e577825+0x4529b)
    #8 _$LT$futures..stream..for_each..ForEach$LT$S$C$$u20$F$C$$u20$U$GT$$u20$as$u20$futures..future..Future$GT$::poll::h6ac1d47ed7a3432b /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/stream/for_each.rs:46 (hammer-c57977986e577825+0x1b61f)
    #9 _$LT$hammer..hammer..Counted$LT$T$GT$$u20$as$u20$futures..future..Future$GT$::poll::h33067f13db245552 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:30 (hammer-c57977986e577825+0x44b9c)
    #10 _$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hc8b313ab120e092b /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/future/mod.rs:113 (hammer-c57977986e577825+0x10e6ec)
    #11 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::poll_future_notify::_$u7b$$u7b$closure$u7d$$u7d$::h5f9cdcc02ea69798 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:289 (hammer-c57977986e577825+0x135388)
    #12 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::enter::_$u7b$$u7b$closure$u7d$$u7d$::h26a02a63cfad5f45 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:363 (hammer-c57977986e577825+0x1358db)
    #13 futures::task_impl::std::set::hf984587eeb93b3cc /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/std/mod.rs:78 (hammer-c57977986e577825+0x12fa42)
    #14 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::enter::ha149ce9f75c55bd4 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:363 (hammer-c57977986e577825+0x13583e)
    #15 _$LT$futures..task_impl..Spawn$LT$T$GT$$GT$::poll_future_notify::h330eeec02baeeb92 /home/travis/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.21/src/task_impl/mod.rs:289 (hammer-c57977986e577825+0x135266)
    #16 tokio_threadpool::task::TaskFuture::poll::h3ee729c6b70c209a /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:292 (hammer-c57977986e577825+0x105cac)
    #17 tokio_threadpool::task::Task::run::_$u7b$$u7b$closure$u7d$$u7d$::hb737ac194cc5977f /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:165 (hammer-c57977986e577825+0xff5ad)
    #18 core::ops::function::FnOnce::call_once::h187d6bb9aa1b58ff /checkout/src/libcore/ops/function.rs:223 (hammer-c57977986e577825+0xf9710)
    #19 _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h610e9aa008480940 /checkout/src/libstd/panic.rs:296 (hammer-c57977986e577825+0x1130cf)
    #20 std::panicking::try::do_call::hbb9f64052a3d1fa3 /checkout/src/libstd/panicking.rs:306 (hammer-c57977986e577825+0x14486b)
    #21 __rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:102 (hammer-c57977986e577825+0x19b1ee)
    #22 std::panic::catch_unwind::hf8936694dbbda59c /checkout/src/libstd/panic.rs:361 (hammer-c57977986e577825+0x11323c)
    #23 tokio_threadpool::task::Task::run::hc5dc9f09396ef157 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/task/mod.rs:151 (hammer-c57977986e577825+0x104e9c)
    #24 tokio_threadpool::worker::Worker::run_task2::heb3e1ea1d9db08ed /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:520 (hammer-c57977986e577825+0x1173d3)
    #25 tokio_threadpool::worker::Worker::run_task::h61de036a84873436 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:420 (hammer-c57977986e577825+0x116ba9)
    #26 tokio_threadpool::worker::Worker::try_run_owned_task::hb53b2ba14f485cdd /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:360 (hammer-c57977986e577825+0x115e9b)
    #27 tokio_threadpool::worker::Worker::try_run_task::hc8fd1c0e2aa76a06 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:289 (hammer-c57977986e577825+0x1154e8)
    #28 tokio_threadpool::worker::Worker::run::h42f2a13700e9f53e /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:235 (hammer-c57977986e577825+0x11529e)
    #29 tokio_threadpool::worker::Worker::do_run::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h96ae7e3abaf68f25 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:119 (hammer-c57977986e577825+0x147307)
    #30 tokio_executor::global::with_default::_$u7b$$u7b$closure$u7d$$u7d$::hb899c5ed9bc1970a /home/travis/build/tokio-rs/tokio/tokio-executor/src/global.rs:176 (hammer-c57977986e577825+0x13bdad)
    #31 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::ha5d64a1ab361f9ba /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b604)
    #32 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::haa2dd0dbde5265ab /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a94a)
    #33 tokio_executor::global::with_default::h9d1a7d8dbb9a0ee8 /home/travis/build/tokio-rs/tokio/tokio-executor/src/global.rs:150 (hammer-c57977986e577825+0x13bc0f)
    #34 tokio_threadpool::worker::Worker::do_run::_$u7b$$u7b$closure$u7d$$u7d$::h4065f71bfa34f9e7 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:115 (hammer-c57977986e577825+0x1474c8)
    #35 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::h5ad0b12fa0a89763 /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b3a3)
    #36 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h32301f38dfac8d20 /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a648)
    #37 tokio_threadpool::worker::Worker::do_run::h9986b07afd6e6570 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:106 (hammer-c57977986e577825+0x115097)
    #38 tokio_threadpool::pool::Pool::spawn_thread::_$u7b$$u7b$closure$u7d$$u7d$::h692f79d8ec032f46 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:398 (hammer-c57977986e577825+0x148eb2)
    #39 std::sys_common::backtrace::__rust_begin_short_backtrace::h01e0e0650fddd788 /checkout/src/libstd/sys_common/backtrace.rs:136 (hammer-c57977986e577825+0x149c1f)
    #40 std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h9d6ddc09ca694ab2 /checkout/src/libstd/thread/mod.rs:406 (hammer-c57977986e577825+0x13ca03)
    #41 _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::ha5fadba8ebb4d993 /checkout/src/libstd/panic.rs:296 (hammer-c57977986e577825+0x11313f)
    #42 std::panicking::try::do_call::h35832c6ed3d077f8 /checkout/src/libstd/panicking.rs:306 (hammer-c57977986e577825+0x1447c8)
    #43 __rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:102 (hammer-c57977986e577825+0x19b1ee)
    #44 std::panic::catch_unwind::h4f6590405ca70495 /checkout/src/libstd/panic.rs:361 (hammer-c57977986e577825+0x1131bc)
    #45 std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::h9086deeaab7af644 /checkout/src/libstd/thread/mod.rs:405 (hammer-c57977986e577825+0x13c7d0)
    #46 _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h1832befc0e5ed971 /checkout/src/liballoc/boxed.rs:783 (hammer-c57977986e577825+0x13ca77)
    #47 _$LT$alloc..boxed..Box$LT$alloc..boxed..FnBox$LT$A$C$$u20$Output$u3d$R$GT$$u20$$u2b$$u20$$u27$a$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::hd924cfcad138b2dd /checkout/src/liballoc/boxed.rs:793 (hammer-c57977986e577825+0x1906f7)
    #48 std::sys_common::thread::start_thread::h3a263f619eb7375b /checkout/src/libstd/sys_common/thread.rs:24 (hammer-c57977986e577825+0x1906f7)
  Thread T19 (tid=37994, running) created by thread T1 at:
    #0 pthread_create /checkout/src/libcompiler_builtins/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (hammer-c57977986e577825+0x4d2a6)
    #1 std::sys::unix::thread::Thread::new::he5ce8706496dfcd2 /checkout/src/libstd/sys/unix/thread.rs:78 (hammer-c57977986e577825+0x1832f9)
    #2 tokio_threadpool::pool::Pool::spawn_thread::hff5239497e631c9c /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:380 (hammer-c57977986e577825+0x11e368)
    #3 tokio_threadpool::pool::Pool::submit_to_external::h6944433290a53d91 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:328 (hammer-c57977986e577825+0x11d6e4)
    #4 tokio_threadpool::pool::Pool::submit_external::h3783cdd23519a2e7 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:304 (hammer-c57977986e577825+0x11cfef)
    #5 tokio_threadpool::pool::Pool::submit::_$u7b$$u7b$closure$u7d$$u7d$::h830d8c681255f053 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:287 (hammer-c57977986e577825+0x148617)
    #6 tokio_threadpool::worker::Worker::with_current::_$u7b$$u7b$closure$u7d$$u7d$::hac6f7b867f2cb3a0 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:135 (hammer-c57977986e577825+0x1476bb)
    #7 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::h384dc84a000e44c1 /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b109)
    #8 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h962936840c1528c9 /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a89a)
    #9 tokio_threadpool::worker::Worker::with_current::h5bf3067078fbcd34 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:131 (hammer-c57977986e577825+0x1475b9)
    #10 tokio_threadpool::pool::Pool::submit::hd1cd6a9d64c5c7ea /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:272 (hammer-c57977986e577825+0x11c9f8)
    #11 _$LT$$RF$$u27$a$u20$tokio_threadpool..sender..Sender$u20$as$u20$tokio_executor..Executor$GT$::spawn::h32b12e3d26f5095d /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:173 (hammer-c57977986e577825+0xf956f)
    #12 tokio_threadpool::sender::Sender::spawn::he5446b8adab58cef /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:87 (hammer-c57977986e577825+0x24f74)
    #13 tokio_threadpool::thread_pool::ThreadPool::spawn::hb646213754578c25 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/thread_pool.rs:64 (hammer-c57977986e577825+0x43ca7)
    #14 hammer::hammer::hc9f00956ee3ac9e9 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:87 (hammer-c57977986e577825+0x29006)
    #15 hammer::__test::TESTS::_$u7b$$u7b$closure$u7d$$u7d$::h21abcb5b42c22027 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:13 (hammer-c57977986e577825+0x2bb04)
    #16 core::ops::function::FnOnce::call_once::h7bb843658796ab47 /checkout/src/libcore/ops/function.rs:223 (hammer-c57977986e577825+0x1cc29)
    #17 test::run_test::_$u7b$$u7b$closure$u7d$$u7d$::h648e6b00b9b34d4c /checkout/src/libtest/lib.rs:1453 (hammer-c57977986e577825+0xbb0b1)
    #18 core::ops::function::FnOnce::call_once::hab331e63f4b9d590 /checkout/src/libcore/ops/function.rs:223 (hammer-c57977986e577825+0xbb0b1)
    #19 _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h8c9806229337e6a4 /checkout/src/liballoc/boxed.rs:783 (hammer-c57977986e577825+0xbb0b1)
  Thread T18 (tid=37993, running) created by thread T1 at:
    #0 pthread_create /checkout/src/libcompiler_builtins/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (hammer-c57977986e577825+0x4d2a6)
    #1 std::sys::unix::thread::Thread::new::he5ce8706496dfcd2 /checkout/src/libstd/sys/unix/thread.rs:78 (hammer-c57977986e577825+0x1832f9)
    #2 tokio_threadpool::pool::Pool::spawn_thread::hff5239497e631c9c /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:380 (hammer-c57977986e577825+0x11e368)
    #3 tokio_threadpool::pool::Pool::submit_to_external::h6944433290a53d91 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:328 (hammer-c57977986e577825+0x11d6e4)
    #4 tokio_threadpool::pool::Pool::submit_external::h3783cdd23519a2e7 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:304 (hammer-c57977986e577825+0x11cfef)
    #5 tokio_threadpool::pool::Pool::submit::_$u7b$$u7b$closure$u7d$$u7d$::h830d8c681255f053 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:287 (hammer-c57977986e577825+0x148617)
    #6 tokio_threadpool::worker::Worker::with_current::_$u7b$$u7b$closure$u7d$$u7d$::hac6f7b867f2cb3a0 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:135 (hammer-c57977986e577825+0x1476bb)
    #7 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::h384dc84a000e44c1 /checkout/src/libstd/thread/local.rs:290 (hammer-c57977986e577825+0x13b109)
    #8 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h962936840c1528c9 /checkout/src/libstd/thread/local.rs:244 (hammer-c57977986e577825+0x13a89a)
    #9 tokio_threadpool::worker::Worker::with_current::h5bf3067078fbcd34 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/worker/mod.rs:131 (hammer-c57977986e577825+0x1475b9)
    #10 tokio_threadpool::pool::Pool::submit::hd1cd6a9d64c5c7ea /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/pool/mod.rs:272 (hammer-c57977986e577825+0x11c9f8)
    #11 _$LT$$RF$$u27$a$u20$tokio_threadpool..sender..Sender$u20$as$u20$tokio_executor..Executor$GT$::spawn::h32b12e3d26f5095d /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:173 (hammer-c57977986e577825+0xf956f)
    #12 tokio_threadpool::sender::Sender::spawn::h4d3ab4e394b134c0 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/sender.rs:87 (hammer-c57977986e577825+0x24c5d)
    #13 tokio_threadpool::thread_pool::ThreadPool::spawn::h9596a22fb8d14e46 /home/travis/build/tokio-rs/tokio/tokio-threadpool/src/thread_pool.rs:64 (hammer-c57977986e577825+0x43b97)
    #14 hammer::hammer::hc9f00956ee3ac9e9 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:50 (hammer-c57977986e577825+0x28d3f)
    #15 hammer::__test::TESTS::_$u7b$$u7b$closure$u7d$$u7d$::h21abcb5b42c22027 /home/travis/build/tokio-rs/tokio/tokio-threadpool/tests/hammer.rs:13 (hammer-c57977986e577825+0x2bb04)
    #16 core::ops::function::FnOnce::call_once::h7bb843658796ab47 /checkout/src/libcore/ops/function.rs:223 (hammer-c57977986e577825+0x1cc29)
    #17 test::run_test::_$u7b$$u7b$closure$u7d$$u7d$::h648e6b00b9b34d4c /checkout/src/libtest/lib.rs:1453 (hammer-c57977986e577825+0xbb0b1)
    #18 core::ops::function::FnOnce::call_once::hab331e63f4b9d590 /checkout/src/libcore/ops/function.rs:223 (hammer-c57977986e577825+0xbb0b1)
    #19 _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h8c9806229337e6a4 /checkout/src/liballoc/boxed.rs:783 (hammer-c57977986e577825+0xbb0b1)
SUMMARY: ThreadSanitizer: data race /checkout/src/libcore/sync/atomic.rs:1601 in core::sync::atomic::atomic_add::h39d213ff06ee7936

@carllerche carllerche mentioned this issue Apr 30, 2018
Merged
@ghost
Copy link

ghost commented May 4, 2018

I'll give this issue a shot!

@carllerche carllerche added the E-help-wanted Call for participation: Help is requested to fix this issue. label May 11, 2018
@carllerche carllerche assigned ghost Jun 15, 2018
This was referenced Jun 18, 2018
Merged
Merged
carllerche added a commit that referenced this issue Jun 18, 2018
@carllerche
Enable backtraces in CI & disable TSAN (#436)
85cf47d 
This PR enables backtraces when running tests and disables tsan for the thread pool.

The thread sanitizer was generating too many false positives. Once #329 lands, then it can
be re-enabled.
@ghost
Copy link

ghost commented Jul 2, 2018

It should be noted that even with crossbeam-rs/crossbeam-deque#13 we'd need something like this in ci/tsan:

# The epoch-based GC uses fences.
race:crossbeam_epoch

# Push and steal operations in crossbeam-deque may cause data races, but such
# data races are safe. If a data race happens, the value read by `steal` is
# forgotten and the steal operation is then retried.
race:crossbeam_deque*push
race:crossbeam_deque*steal

First, crossbeam-epoch uses fences so we need to silence tsan errors coming from it.

Second, the Chase-Lev (even as the cicbuf variant) algorithm is inherently data-racy. The steal operation first reads data from the buffer and then checks whether something changed in the meantime. If so, the read data is discarded (using mem::forget) and the steal operation has to be retried. Put differently, a data race might happen, but we never follow up on it so UB doesn't happen.

@carllerche
Copy link
Member Author

@stjepang Whitelisting those two would be fine. I mostly am hoping to avoid a fence to make the data being stolen from being visible.

@ghost ghost mentioned this issue Jul 6, 2018
Merged
@Darksonn Darksonn removed the E-help-wanted Call for participation: Help is requested to fix this issue. label Sep 25, 2020
@taiki-e taiki-e mentioned this issue Nov 3, 2020
Closed
@taiki-e taiki-e mentioned this issue Feb 20, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@carllerche @Darksonn