Skip to content

Commit

Permalink
Use initial exec TLS model for all thread local variables from thread…
Browse files Browse the repository at this point in the history
…_cache.cc

This avoids a deadlock when a library which is being dlopen'ed creates
as part of its static constructors a thread which quickly need to call
malloc. We are still in the dlopen call (so with some internal glibc
mutex taken) when the thread executes code and later needs to call
malloc which in term calls tls_get_addr_tail, which wait for the dlopen
mutex to be unlocked. If later the dlopen'ing thread also calls malloc
as part of its constructors, we are in a deadlock.

Fix is similar to
gperftools/gperftools@7852eeb

Stack of the dlopening thread:
    #0  0x00007fd5406ca93c in __lll_lock_wait () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libpthread.so.0
    #1  0x00007fd5406c45a5 in pthread_mutex_lock () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libpthread.so.0
    ... proprietary code in the stack
    envoyproxy#9  0x00007fd5074f0367 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at src/ClientImpl.cpp:15
    envoyproxy#10 0x00007fd5074f06d7 in _GLOBAL__sub_I_ClientImpl.cpp(void) () at src/ClientImpl.cpp:85
    envoyproxy#11 0x00007fd50757aa46 in __do_global_ctors_aux ()
    envoyproxy#12 0x00007fd5073e985f in _init () from ...
    envoyproxy#13 0x00007fd53bf9dec8 in ?? () from ...
    envoyproxy#14 0x00007fd54d637a5d in call_init.part () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2
    envoyproxy#15 0x00007fd54d637bab in _dl_init () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2
    envoyproxy#16 0x00007fd54d63c160 in dl_open_worker () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2
    envoyproxy#17 0x00007fd54d637944 in _dl_catch_error () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2
    envoyproxy#18 0x00007fd54d63b7d9 in _dl_open () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2
    envoyproxy#19 0x00007fd54d61f2b9 in dlopen_doit () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libdl.so.2
    envoyproxy#20 0x00007fd54d637944 in _dl_catch_error () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2
    envoyproxy#21 0x00007fd54d61f889 in _dlerror_run () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libdl.so.2
    envoyproxy#22 0x00007fd54d61f351 in dlopen@@GLIBC_2.2.5 () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libdl.so.2

Stack of the newly created thread calling tls_get_addr_tail:
    #0  0x00007fd5406ca93c in __lll_lock_wait () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libpthread.so.0
    #1  0x00007fd5406c4622 in pthread_mutex_lock () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libpthread.so.0
    envoyproxy#2  0x00007fd54d63a2ed in tls_get_addr_tail () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2
    envoyproxy#3  0x00007fd53fee877d in tcmalloc::ThreadCache::CreateCacheIfNecessary () at src/thread_cache.cc:344
    envoyproxy#4  0x00007fd53fecb4ab in tcmalloc::ThreadCache::GetCache () at src/thread_cache.h:437
    envoyproxy#5  0x00007fd53fefeccb in (anonymous namespace)::do_malloc (size=56) at src/tcmalloc.cc:1354
    envoyproxy#6  tcmalloc::do_allocate_full<tcmalloc::cpp_throw_oom> (size=56) at src/tcmalloc.cc:1762
    envoyproxy#7  tcmalloc::allocate_full_cpp_throw_oom (size=56) at src/tcmalloc.cc:1776
    envoyproxy#8  0x00007fd53ff01b80 in tcmalloc::dispatch_allocate_full<tcmalloc::cpp_throw_oom> (size=56) at src/tcmalloc.cc:1785
    envoyproxy#9  malloc_fast_path<tcmalloc::cpp_throw_oom> (size=56) at src/tcmalloc.cc:1845
    envoyproxy#10 tc_new (size=56) at src/tcmalloc.cc:1980
    ... proprietary code in the stack
    envoyproxy#26 0x00007fd5406c1ef4 in start_thread () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libpthread.so.0
    envoyproxy#27 0x00007fd5403ba01d in clone () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libc.so.6
  • Loading branch information
Romain-Geissler-1A authored and alk committed Feb 10, 2019
1 parent 8f9a873 commit 1e36ed7
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/thread_cache.cc
Original file line number Diff line number Diff line change
Expand Up @@ -334,7 +334,7 @@ ThreadCache* ThreadCache::CreateCacheIfNecessary() {

bool seach_condition = true;
#ifdef HAVE_TLS
static __thread ThreadCache** current_heap_ptr;
static __thread ThreadCache** current_heap_ptr ATTR_INITIAL_EXEC;
if (tsd_inited_) {
// In most common case we're avoiding expensive linear search
// through all heaps (see below). Working TLS enables faster
Expand Down

0 comments on commit 1e36ed7

Please sign in to comment.