Segmentation fault after wrangler executes its command #1464
Comments
|
I've noticed this too (on Linux). Seems to be common after error messages, but it doesn't always happen. |
|
are you able to reproduce this issue if you build with the command |
Still have a segmentation fault. |
|
It seems the issue goes away if I have a |
|
I could minify the bug to this: fn main() {
std::thread::spawn(move || {
reqwest::blocking::Client::new()
.get("https://crates.io/api/v1/crates/wrangler")
.send().unwrap();
});
// Just so we write some memory to trigger the bug.
let mut v = Vec::new();
for i in 0..100000 {
v.push(i);
}
}Still get the segmentation fault almost all the time. It looks like it might be cause by sfackler/rust-openssl#1293. |
|
That does seem to be the issue doesn't it - nice job tracking it down! If your PR gets merged upstream and a release is cut we'll be sure to update the dependency. Thanks a bunch @orium! |
|
Hey @orium - I'm not able to reproduce this as is. Could you include your |
|
I think I don't have it anymore, but IIRC it only had a dependency on In any case the issue is fixed upstream (sfackler/rust-openssl#1293) and once we get a new release of rust-openssl (which is a transitive dependency of |
|
yes, reproduced with: wait, when writing this comment, i got another error message when just executing Updateokay, this weird
|
|
@codenoid I think the is caused by the same issue, so the upstream fix in rust-openssl should fix that as well. |
|
This is now fixed since we are using openssl v0.10.32, which has a fix for this. |
|
this is still a problem on the latest version of wrangler (despite being on openssl v0.10.32), it can manifest as either a segfault or the panic at client::new() |
|
I tried this hack (sfackler/rust-openssl#1174 (comment)) and it was unsuccessful, I can still replicate a segfault on fully statically linked binary every 1/10 invocations or so. EDIT: backtrace |
|
I just tried running It seems like the segmentation fault is not systematic for me either. It doesn't always happen, about 50% of the time. |
|
I GDB'd the error and this is what I got, not sure what to make out of it: |
|
Here's a backtrace from Starting program: /home/nick/.cargo/bin/wrangler generate --type=rust rate_limit_cloudflare_worker
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
[New Thread 0x7ffff72fa640 (LWP 1109341)]
[New Thread 0x7ffff70f9640 (LWP 1109342)]
⬇️ Installing cargo-generate v0.5.0...
[New Thread 0x7ffff6ef8640 (LWP 1109343)]
[Thread 0x7ffff6ef8640 (LWP 1109343) exited]
[New Thread 0x7ffff6ef8640 (LWP 1109344)]
Thread 1 "wrangler" received signal SIGSEGV, Segmentation fault.
0x00007ffff7a9b2c3 in SSL_get_peer_certificate () from /usr/lib/libssl.so.1.1
(gdb) bt full
#0 0x00007ffff7a9b2c3 in SSL_get_peer_certificate () from /usr/lib/libssl.so.1.1
No symbol table info available.
#1 0x00007ffff7f6fd66 in ?? () from /usr/lib/libcurl.so.4
No symbol table info available.
#2 0x00007ffff7f73f05 in ?? () from /usr/lib/libcurl.so.4
No symbol table info available.
#3 0x00007ffff7f74fd7 in ?? () from /usr/lib/libcurl.so.4
No symbol table info available.
#4 0x00007ffff7f2c149 in ?? () from /usr/lib/libcurl.so.4
No symbol table info available.
#5 0x00007ffff7f45508 in ?? () from /usr/lib/libcurl.so.4
No symbol table info available.
#6 0x00007ffff7f46b06 in curl_multi_perform () from /usr/lib/libcurl.so.4
No symbol table info available.
#7 0x00007ffff7f1ed0c in curl_easy_perform () from /usr/lib/libcurl.so.4
No symbol table info available.
#8 0x0000555555bac2aa in curl::easy::handler::Easy2<H>::perform ()
No symbol table info available.
#9 0x0000555555baadb2 in curl::easy::handle::Transfer::perform ()
No symbol table info available.
#10 0x0000555555ba3162 in binary_install::curl ()
No symbol table info available.
#11 0x0000555555b9f741 in binary_install::Cache::_download ()
No symbol table info available.
#12 0x0000555555b9f32a in binary_install::Cache::download_version ()
No symbol table info available.
#13 0x0000000000000001 in ?? ()
No symbol table info available.
#14 0x00005555565c2180 in ?? ()
No symbol table info available.
#15 0x0000000000000070 in ?? ()
No symbol table info available.
#16 0x00005555565bd970 in ?? ()
No symbol table info available.
#17 0x0000000000000005 in ?? ()
No symbol table info available.
#18 0x0000000000000005 in ?? ()
No symbol table info available.
#19 0x000000000000000e in ?? ()
No symbol table info available.
#20 0x0000555555ab7d3f in wrangler::install::install ()
No symbol table info available.
#21 0x0000555555ab71ab in wrangler::install::install_cargo_generate ()
No symbol table info available.
#22 0x00007fffffffd688 in ?? ()
No symbol table info available.
#23 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) > openssl version
OpenSSL 1.1.1l 24 Aug 2021
> cargo --version
cargo 1.56.0 (4ed5d137b 2021-10-04)
> wrangler --version
wrangler 1.19.5I also confirmed the same behavior in a fresh arch linux install with minimal dependencies ( It also happens even if I build > wrangler/target/debug/wrangler generate --type=rust rate_limit_cloudflare_worker
⬇️ Installing cargo-generate v0.5.0...
fish: Job 1, 'wrangler/target/debug/wrangler…' terminated by signal SIGSEGV (Address boundary error)The server it talks to seems fine: > curl -vvv https://workers.cloudflare.com/get-binary/ashleygwilliams/cargo-generate/v0.5.0/x86_64-unknown-linux-musl.tar.gz
* Trying 104.16.132.9:443...
* Connected to workers.cloudflare.com (104.16.132.9) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
... |
|
I don't understand why cargo isn't just used to pull in I was able to modify diff --git a/src/install/mod.rs b/src/install/mod.rs
index d7418ad..cc6f627 100644
--- a/src/install/mod.rs
+++ b/src/install/mod.rs
@@ -27,9 +27,7 @@ pub fn install_cargo_generate() -> Result<PathBuf> {
let tool_author = "ashleygwilliams";
let is_binary = true;
let version = Version::parse(dependencies::GENERATE_VERSION)?;
- install(tool_name, tool_author, is_binary, version)?
- .binary(tool_name)
- .map_err(|e| anyhow::Error::from(e.compat()))
+ Ok(PathBuf::from(r"/home/nick/.cargo/bin/cargo-generate"))
}
pub fn install_wasm_pack() -> Result<PathBuf> {
@@ -37,9 +35,7 @@ pub fn install_wasm_pack() -> Result<PathBuf> {
let tool_author = "rustwasm";
let is_binary = true;
let version = Version::parse(dependencies::WASM_PACK_VERSION)?;
- install(tool_name, tool_author, is_binary, version)?
- .binary(tool_name)
- .map_err(|e| anyhow!(e.compat()))
+ Ok(PathBuf::from(r"/home/nick/.cargo/bin/wasm-pack"))
}
pub fn install(
That let me run I then got a similar
> ~/wrangler/target/debug/wrangler publish
🌀 Running cargo install -q worker-build && worker-build --release
[INFO]: Checking for the Wasm target...
[INFO]: Compiling to Wasm...
Finished release [optimized] target(s) in 0.05s
[INFO]: Installing wasm-bindgen...
Error: wasm-pack exited with status signal: 11 (core dumped)
Error: Build failed! Status Code: 1
|
|
Downgrading to |
|
Hi all, I'm having a similar issue on Arch Linux. I installed the latest (v1.19.5) version of wrangler via cargo, but when I try to generate a new Rust project, I get this error. I was able to generate a new project when I installed the node version, but then when I try to run a development server, I get the following error: Is there any solution to this? I know that the above issue could be fixed by recompiling my own version with |
|
@xortive here's a backtrace with debug symbols if it helps (it cuts off after the tokio internals because they're not interesting): |
|
Hmm, it looks like OpenSSL has installed its own exit handlers? I wonder if they don't interact well with multi-threading ... there's an open rust issue about this: rust-lang/rust#83994 I also noticed that this is making network requests right at startup, which seems odd - it turns out wrangler is checking for updates in the background: Maybe we can wait to do that until the process exits / only do it if wrangler finishes successfully, so there's no other threads running at the same time? |
|
I guess another alternative is to kill the thread looking for background updates before exiting, but that seems unnecessarily complicated when all it gets us is slightly faster latency when exiting the process. |
|
Yet another alternative is to avoid constructing a |
|
Hello dear colleagues, I've faced core dumps too with both 1.19.5 and #2150 versions. Here my backtrace and what I've run: Sadly, I can't provide more info unless one would ask me for something specific. |
|
Closing as there is a new version of Wrangler built on node.js, which should not get such segmentation faults. If you can reproduce on the new Wrangler, please create a new issue at https://github.com/cloudflare/wrangler2/issues/new/choose. |
🐛 Bug Report
wrangler crashes with a segmentation fault after running the given command. This apparently only happens with release build and not with debug builds. Reproducible with wrangler version 1.10.3 as well as master (as of b601101); also reproducible with both rustc 1.45.0 (5c1f21c3b 2020-07-13) as well as rustc 1.46.0-nightly (346aec9b0 2020-07-11).
Environment
rustc -V: I've tried multiple version, but including the currect stable: rustc 1.45.0 (5c1f21c3b 2020-07-13)node -v: not installedwrangler -V: repoducible in 1.10.3 and master (as of b601101)wrangler.toml: emptySteps to reproduce
You can either
cargo install wranglerfrom crates.io, or in master (as of b601101):Running inside lldb:
Also worth saying that valgrind doesn't see any invalid memory accesses.
The text was updated successfully, but these errors were encountered: