Segfault when used from rusoto and using sync functions #1174

dten · 2019-10-12T16:35:47Z

I'm not sure where to ask about this.

Basically we have a simple program that is using rusoto to query aws. Rusoto's functions return futures but offer a .sync() function to run them in a lazy static runtime (https://docs.rs/rusoto_core/0.41.0/rusoto_core/struct.RusotoFuture.html#method.sync)

roughly 10-30% of the time when the program is finished (ie after the last line of our main) we get this segfault

rusoto has a rustls option and if i switch to that nothing explodes

this is ubuntu 18.04 and open ssl 1.1.1-1ubuntu2.1~

also openssl is only coming from native-tls

[[package]]
name = "native-tls"
version = "0.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
 "lazy_static 1.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
 "libc 0.2.62 (registry+https://github.com/rust-lang/crates.io-index)",
 "log 0.4.8 (registry+https://github.com/rust-lang/crates.io-index)",
 "openssl 0.10.25 (registry+https://github.com/rust-lang/crates.io-index)",
 "openssl-probe 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
 "openssl-sys 0.9.51 (registry+https://github.com/rust-lang/crates.io-index)",
 "schannel 0.1.16 (registry+https://github.com/rust-lang/crates.io-index)",
 "security-framework 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)",
 "security-framework-sys 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)",
 "tempfile 3.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
]

❯ rust-lldb target/debug/check_ecs -c core
(lldb) command script import "/mnt/d/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/etc/lldb_rust_formatters.py"
(lldb) type summary add --no-value --python-function lldb_rust_formatters.print_val -x ".*" --category Rust
(lldb) type category enable Rust
(lldb) target create "target/debug/check_ecs" --core "core"
Core file '/home/david/Documents/Repositories/pascal/ecsthing/core' (x86_64) was loaded.
(lldb) bt
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
* thread #1, name = 'check_ecs', stop reason = signal SIGSEGV
  * frame #0: 0x00001471cc735802 libpthread.so.0`__GI___pthread_rwlock_wrlock + 18
    frame #1: 0x00001471ccf33889 libcrypto.so.1.1`CRYPTO_THREAD_write_lock + 9
    frame #2: 0x00001471ccef4f43 libcrypto.so.1.1`RAND_get_rand_method + 51
    frame #3: 0x00001471ccef5220 libcrypto.so.1.1`RAND_bytes + 16
    frame #4: 0x00001471cd2451be libssl.so.1.1`___lldb_unnamed_symbol139$$libssl.so.1.1 + 302
    frame #5: 0x00001471cd242860 libssl.so.1.1`___lldb_unnamed_symbol114$$libssl.so.1.1 + 2672
    frame #6: 0x00001471cd24c6cc libssl.so.1.1`___lldb_unnamed_symbol201$$libssl.so.1.1 + 92
    frame #7: 0x00001471cd24a7a5 libssl.so.1.1`___lldb_unnamed_symbol182$$libssl.so.1.1 + 197
    frame #8: 0x00001471cd2556bf libssl.so.1.1`SSL_shutdown + 63
    frame #9: 0x000055c0fc3a0f54 check_ecs`openssl::ssl::SslStream$LT$S$GT$::shutdown::h1dc2ace74fbb1c63(self=&0x1471a40116c0) at mod.rs:3428
    frame #10: 0x000055c0fc33766c check_ecs`native_tls::imp::TlsStream$LT$S$GT$::shutdown::hcb3faec2fd8443a6(self=&0x1471a40116c0) at openssl.rs:384
    frame #11: 0x000055c0fc3bc1ec check_ecs`native_tls::TlsStream$LT$S$GT$::shutdown::he0266766074ef79e(self=&0x1471a40116c0) at lib.rs:642
    frame #12: 0x000055c0fc44409c check_ecs`_$LT$hyper_tls..stream..TlsStream$LT$T$GT$$u20$as$u20$tokio_io..async_write..AsyncWrite$GT$::shutdown::hccce62523e08d7a8(self=&0x1471a40116c0) at stream.rs:181
    frame #13: 0x000055c0fc44445a check_ecs`_$LT$hyper_tls..stream..MaybeHttpsStream$LT$T$GT$$u20$as$u20$tokio_io..async_write..AsyncWrite$GT$::shutdown::h8a74a8ef58ce504b(self=&0x1471a40116b8) at stream.rs:110
    frame #14: 0x000055c0fc3978dd check_ecs`hyper::proto::h1::conn::Conn$LT$I$C$B$C$T$GT$::shutdown::h933c4140b3416588(self=&0x1471a40116b8) at conn.rs:597
    frame #15: 0x000055c0fc3c8e6f check_ecs`hyper::proto::h1::dispatch::Dispatcher$LT$D$C$Bs$C$I$C$T$GT$::poll_inner::h5f222f69463d4bf8(self=&0x1471a40116b8, should_shutdown=true) at dispatch.rs:113
    frame #16: 0x000055c0fc3c867a check_ecs`hyper::proto::h1::dispatch::Dispatcher$LT$D$C$Bs$C$I$C$T$GT$::poll_catch::h45642cd6cb54afd1(self=&0x1471a40116b8, should_shutdown=true) at dispatch.rs:93
    frame #17: 0x000055c0fc3bbef1 check_ecs`_$LT$hyper..proto..h1..dispatch..Dispatcher$LT$D$C$Bs$C$I$C$T$GT$$u20$as$u20$futures..future..Future$GT$::poll::h3cb12da198435139(self=&0x1471a40116b8) at dispatch.rs:374
    frame #18: 0x000055c0fc3f2048 check_ecs`_$LT$futures..future..either..Either$LT$A$C$B$GT$$u20$as$u20$futures..future..Future$GT$::poll::ha924a5c792a593d5(self=&0x1471a40116b0) at either.rs:35
    frame #19: 0x000055c0fc3d07db check_ecs`futures::future::option::_$LT$impl$u20$futures..future..Future$u20$for$u20$core..option..Option$LT$F$GT$$GT$::poll::hb3b8fa69c08c5aff(self=&0x1471a40116b0) at option.rs:12
    frame #20: 0x000055c0fc359632 check_ecs`_$LT$hyper..client..conn..Connection$LT$T$C$B$GT$$u20$as$u20$futures..future..Future$GT$::poll::h3df0ee679a4b462e(self=&0x1471a40116b0) at conn.rs:414
    frame #21: 0x000055c0fc4567b7 check_ecs`_$LT$futures..future..map_err..MapErr$LT$A$C$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h46d743be8834af05(self=&0x1471a40116b0) at map_err.rs:30
    frame #22: 0x000055c0fcc680e8 check_ecs`_$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h6503a097a67e6d54(self=&0x1471a4021098) at mod.rs:113
    frame #23: 0x000055c0fca4bb54 check_ecs`futures::task_impl::Spawn$LT$T$GT$::poll_future_notify::_$u7b$$u7b$closure$u7d$$u7d$::h691311cb7e01ce71((null)=closure {

}, f=&0x1471a4021098) at mod.rs:329
    frame #24: 0x000055c0fca4be14 check_ecs`futures::task_impl::Spawn$LT$T$GT$::enter::_$u7b$$u7b$closure$u7d$$u7d$::hcd90395ce3f59bf7 at mod.rs:399
    frame #25: 0x000055c0fca4c214 check_ecs`futures::task_impl::std::set::hcc10c26ea8def4a0(task=&0x1471caf14030, f=closure(closure {

}, &0x1471caf14090)) at mod.rs:83
    frame #26: 0x000055c0fca4bc4f check_ecs`futures::task_impl::Spawn$LT$T$GT$::enter::h63fbd78318af2a1d(self=&0x1471a4021060, unpark=<unavailable>, f=closure {

}) at mod.rs:399
    frame #27: 0x000055c0fca4b925 check_ecs`futures::task_impl::Spawn$LT$T$GT$::poll_fn_notify::h3e92af2aa171f941(self=&0x1471a4021060, notify=&0x1471caf14b00, id=22478315458624, f=closure {

}) at mod.rs:291
    frame #28: 0x000055c0fca4baf8 check_ecs`futures::task_impl::Spawn$LT$T$GT$::poll_future_notify::h7675be8999dbdec2(self=&0x1471a4021060, notify=&0x1471caf14b00, id=22478315458624) at mod.rs:329
    frame #29: 0x000055c0fca4e6f2 check_ecs`tokio_threadpool::task::Task::run::_$u7b$$u7b$closure$u7d$$u7d$::h304af29ce3166538 at mod.rs:145
    frame #30: 0x000055c0fca60189 check_ecs`core::ops::function::FnOnce::call_once::h0cba14d4a870672d((null)=<unavailable>, (null)=<unavailable>) at function.rs:235
    frame #31: 0x000055c0fca43009 check_ecs`_$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h5c01b59bf6fc6a9e(self=<unavailable>, _args=<unavailable>) at panic.rs:315
    frame #32: 0x000055c0fca454a9 check_ecs`std::panicking::try::do_call::h16216a1c5d3f8bb2(data=&0x1471caf14310) at panicking.rs:296
    frame #33: 0x000055c0fce9630a check_ecs`__rust_maybe_catch_panic at lib.rs:80
    frame #34: 0x000055c0fca45310 check_ecs`std::panicking::try::hcfe043c59d016e43(f=<unavailable>) at panicking.rs:275
    frame #35: 0x000055c0fca45061 check_ecs`std::panic::catch_unwind::h14135d84175a47c2(f=<unavailable>) at panic.rs:394
    frame #36: 0x000055c0fca4df8e check_ecs`tokio_threadpool::task::Task::run::hde28890093665c93(self=&0x1471a4021040, unpark=&0x1471caf14b00) at mod.rs:130
    frame #37: 0x000055c0fca674f1 check_ecs`tokio_threadpool::worker::Worker::run_task2::h72ff0bc4ce877c40(self=&0x1471caf15798, task=&0x1471caf148f8, notify=&0x1471caf14b00) at mod.rs:567
    frame #38: 0x000055c0fca66d64 check_ecs`tokio_threadpool::worker::Worker::run_task::hfc2a171f0309ba37(self=&0x1471caf15798, task=Arc<tokio_threadpool::task::Task> {     
...

The text was updated successfully, but these errors were encountered:

sfackler · 2019-10-12T17:27:12Z

This sounds like openssl/openssl#6214 - there's not really anything this library can do about it. You can use libc::__exit() to bypass the registered atexit callbacks if you want.

dten · 2019-10-13T07:19:18Z

Ooof thanks. So all threads must be done before main ends? Presumably this is a problem for any lazy static tokio runtime that uses openssl?

sfackler · 2019-10-13T12:07:53Z

Yeah, it's not great :(

I just went the __exit route personally for a service I work on.

dten · 2019-10-13T12:56:16Z

where should that be called? the main thread before exit?

sfackler · 2019-10-13T13:00:01Z

Yep. __exit() is just like https://doc.rust-lang.org/std/process/fn.exit.html except that it doesn't run atexit handlers.

dten · 2019-10-13T16:26:44Z

Interesting thanks

NOTE: During work on this patch, the application faulted very early in processing. This may have been due to a library issue around libssl setting an `_atexit()` handler that conflicts with the rust threading model. (see sfackler/rust-openssl#1174 and openssl/openssl#6214 for possibly related issues) Closes #645

jyn514 · 2021-12-07T21:55:52Z

openssl/openssl#6214 (comment):

I think it would be good to have possibility at runtime to disable calling OPENSSL_cleanup at exit. Maybe it could be turned on or off by using some flag to OPENSSL_init_crypto.

Seems this was already added in openssl 1.1.1 as OPENSSL_INIT_NO_ATEXIT.

@sfackler what do you think about setting that env var when building openssl? I'm not sure if rust-openssl uses the system shared library or builds from source, but it would be nice to at least have an option to not have exit() segfault ...

sfackler · 2021-12-07T22:07:05Z

That value is set at runtime, not compile time: https://github.com/sfackler/rust-openssl/blob/master/openssl-sys/src/lib.rs#L111

jyn514 · 2021-12-07T22:16:40Z

@sfackler hmm I'm not sure I follow - do users of openssl-sys have to call init() explicitly? I don't know why the atexit handlers would still be registered when you're passing OPENSSL_INIT_NO_ATEXIT.

sfackler · 2021-12-07T22:26:29Z

It is called by the openssl crate automatically.

dten mentioned this issue Oct 13, 2019

Simple EC2 client request leads to segfault rusoto/rusoto#1455

Closed

sfackler closed this as completed Oct 13, 2019

jrconlin mentioned this issue Jul 15, 2020

feat: re-enable offset optimization mozilla-services/syncstorage-rs#724

Closed

xortive mentioned this issue Feb 24, 2021

Segmentation fault after wrangler executes its command cloudflare/wrangler-legacy#1464

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segfault when used from rusoto and using sync functions #1174

Segfault when used from rusoto and using sync functions #1174

dten commented Oct 12, 2019 •

edited

Loading

sfackler commented Oct 12, 2019

dten commented Oct 13, 2019

sfackler commented Oct 13, 2019

dten commented Oct 13, 2019

sfackler commented Oct 13, 2019

dten commented Oct 13, 2019

jyn514 commented Dec 7, 2021 •

edited

Loading

sfackler commented Dec 7, 2021

jyn514 commented Dec 7, 2021

sfackler commented Dec 7, 2021

Segfault when used from rusoto and using sync functions #1174

Segfault when used from rusoto and using sync functions #1174

Comments

dten commented Oct 12, 2019 • edited Loading

sfackler commented Oct 12, 2019

dten commented Oct 13, 2019

sfackler commented Oct 13, 2019

dten commented Oct 13, 2019

sfackler commented Oct 13, 2019

dten commented Oct 13, 2019

jyn514 commented Dec 7, 2021 • edited Loading

sfackler commented Dec 7, 2021

jyn514 commented Dec 7, 2021

sfackler commented Dec 7, 2021

dten commented Oct 12, 2019 •

edited

Loading

jyn514 commented Dec 7, 2021 •

edited

Loading