Panic in call to indicatif, causes 100% CPU on MacOS with Homebrew binary

[fiadliel]

When trying to install gcloud with rtx install gcloud, or rtx install, or variations thereof, rtx would start using 100% CPU, and would hang indefinitely.

When running a debug build, I got a backtrace:

✗ RUST_BACKTRACE=1 ~/src/github.com/jdx/rtx/target/debug/rtx i gcloud
The application panicked (crashed).
Message:  attempt to subtract with overflow
Location: /Users/xxx/.local/share/rtx/installs/rust/1.72.0/registry/src/index.crates.io-6f17d22bba15001f/indicatif-0.17.6/src/draw_target.rs:534

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 13 frames hidden ⋮
  14: core::panicking::panic::h4d2ee9cac0d6e9c1
      at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:117
  15: indicatif::draw_target::DrawState::draw_to_term::h6ec0c9392b2d466b
      at /Users/xxx/.local/share/rtx/installs/rust/1.72.0/registry/src/index.crates.io-6f17d22bba15001f/indicatif-0.17.6/src/draw_target.rs:534
  16: indicatif::draw_target::Drawable::draw::h155b24b3f5f1f90e
      at /Users/xxx/.local/share/rtx/installs/rust/1.72.0/registry/src/index.crates.io-6f17d22bba15001f/indicatif-0.17.6/src/draw_target.rs:328
  17: indicatif::multi::MultiState::draw::hc91b92bf5d99b992
      at /Users/xxx/.local/share/rtx/installs/rust/1.72.0/registry/src/index.crates.io-6f17d22bba15001f/indicatif-0.17.6/src/multi.rs:337
  18: indicatif::draw_target::Drawable::draw::h155b24b3f5f1f90e
      at /Users/xxx/.local/share/rtx/installs/rust/1.72.0/registry/src/index.crates.io-6f17d22bba15001f/indicatif-0.17.6/src/draw_target.rs:334
  19: indicatif::state::BarState::println::h1b4955583367a2cc
      at /Users/xxx/.local/share/rtx/installs/rust/1.72.0/registry/src/index.crates.io-6f17d22bba15001f/indicatif-0.17.6/src/state.rs:160
  20: indicatif::progress_bar::ProgressBar::println::h0851b6baa868f979
      at /Users/xxx/.local/share/rtx/installs/rust/1.72.0/registry/src/index.crates.io-6f17d22bba15001f/indicatif-0.17.6/src/progress_bar.rs:245
  21: rtx::ui::progress_report::ProgressReport::println::hb2abab2fce7e38fb
      at /Users/xxx/src/github.com/jdx/rtx/src/ui/progress_report.rs:87
  22: rtx::cmd::CmdLineRunner::on_stderr::h25584fadf773198e
      at /Users/xxx/src/github.com/jdx/rtx/src/cmd.rs:246
  23: rtx::cmd::CmdLineRunner::execute::h7432c2b155b03a1c
      at /Users/xxx/src/github.com/jdx/rtx/src/cmd.rs:204
  24: rtx::plugins::script_manager::ScriptManager::run_by_line::hc3e9f457a1372739
      at /Users/xxx/src/github.com/jdx/rtx/src/plugins/script_manager.rs:177
  25: <rtx::plugins::external_plugin::ExternalPlugin as rtx::plugins::Plugin>::install_version::{{closure}}::h2899de540dd843bc
      at /Users/xxx/src/github.com/jdx/rtx/src/plugins/external_plugin.rs:607
  26: <rtx::plugins::external_plugin::ExternalPlugin as rtx::plugins::Plugin>::install_version::hf2dfec8d0d96b1d6
      at /Users/xxx/src/github.com/jdx/rtx/src/plugins/external_plugin.rs:613
  27: rtx::tool::Tool::install_version::hf1be825d2516bc9a
      at /Users/xxx/src/github.com/jdx/rtx/src/tool.rs:222
  28: rtx::toolset::Toolset::install_versions::{{closure}}::{{closure}}::{{closure}}::h26140f26779b135f
      at /Users/xxx/src/github.com/jdx/rtx/src/toolset/mod.rs:169
                                ⋮ 13 frames hidden ⋮

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
Run with RUST_BACKTRACE=full to include source snippets.
The application panicked (crashed).
Message:  called `Result::unwrap()` on an `Err` value: PoisonError { .. }
Location: /Users/xxx/.local/share/rtx/installs/rust/1.72.0/registry/src/index.crates.io-6f17d22bba15001f/indicatif-0.17.6/src/draw_target.rs:132

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 14 frames hidden ⋮
  15: core::result::Result<T,E>::unwrap::hdf144fe9ced6a28e
      at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/result.rs:1076
  16: indicatif::draw_target::ProgressDrawTarget::width::h79beb90d7d5815ea
      at /Users/xxx/.local/share/rtx/installs/rust/1.72.0/registry/src/index.crates.io-6f17d22bba15001f/indicatif-0.17.6/src/draw_target.rs:132
  17: indicatif::state::BarState::draw::h82ca7d84acc373dc
      at /Users/xxx/.local/share/rtx/installs/rust/1.72.0/registry/src/index.crates.io-6f17d22bba15001f/indicatif-0.17.6/src/state.rs:178
  18: indicatif::state::BarState::finish_using_style::h88c418ab9aa71370
      at /Users/xxx/.local/share/rtx/installs/rust/1.72.0/registry/src/index.crates.io-6f17d22bba15001f/indicatif-0.17.6/src/state.rs:68
  19: <indicatif::state::BarState as core::ops::drop::Drop>::drop::h7d895a60e5424e3f
      at /Users/xxx/.local/share/rtx/installs/rust/1.72.0/registry/src/index.crates.io-6f17d22bba15001f/indicatif-0.17.6/src/state.rs:209
  20: core::ptr::drop_in_place<indicatif::state::BarState>::h759606492e02d576
      at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ptr/mod.rs:497
  21: core::ptr::drop_in_place<core::cell::UnsafeCell<indicatif::state::BarState>>::h771d3bba237f1bbd
      at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ptr/mod.rs:497
  22: core::ptr::drop_in_place<std::sync::mutex::Mutex<indicatif::state::BarState>>::h1de5500c31bf2f56
      at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ptr/mod.rs:497
  23: alloc::sync::Arc<T>::drop_slow::hd4504863b8c5d746
      at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/alloc/src/sync.rs:1263
  24: <alloc::sync::Arc<T> as core::ops::drop::Drop>::drop::h8663806776597db7
      at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/alloc/src/sync.rs:1897
  25: core::ptr::drop_in_place<alloc::sync::Arc<std::sync::mutex::Mutex<indicatif::state::BarState>>>::h6f2436afd44f7925
      at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ptr/mod.rs:497
  26: core::ptr::drop_in_place<indicatif::progress_bar::ProgressBar>::h5436d444cf259cb4
      at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ptr/mod.rs:497
  27: core::ptr::drop_in_place<core::option::Option<indicatif::progress_bar::ProgressBar>>::h9b6c2e55dfd0a9c2
      at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ptr/mod.rs:497
  28: core::ptr::drop_in_place<rtx::ui::progress_report::ProgressReport>::h7c835d65e5fb74a8
      at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ptr/mod.rs:497
  29: rtx::toolset::Toolset::install_versions::{{closure}}::{{closure}}::{{closure}}::h26140f26779b135f
      at /Users/xxx/src/github.com/jdx/rtx/src/toolset/mod.rs:170
                                ⋮ 13 frames hidden ⋮

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
Run with RUST_BACKTRACE=full to include source snippets.
The application panicked (crashed).
Message:  panic in a function that cannot unwind
Location: library/core/src/panicking.rs:126

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 12 frames hidden ⋮
  13: core::panicking::panic_nounwind_fmt::h6e49aceea1598c00
      at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:96
  14: core::panicking::panic_nounwind::h4420bc9d05a9c1db
      at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:126
  15: core::panicking::panic_cannot_unwind::hcd3e1319ebcd2ce0
      at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:189
  16: rtx::toolset::Toolset::install_versions::{{closure}}::{{closure}}::{{closure}}::h26140f26779b135f
      at /Users/xxx/src/github.com/jdx/rtx/src/toolset/mod.rs:163
                                ⋮ 13 frames hidden ⋮

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
Run with RUST_BACKTRACE=full to include source snippets.
thread caused non-unwinding panic. aborting.

The problem looks like console-rs/indicatif#582 upstream, which has no solution yet.

** To reproduce
To reproduce, it may happen when trying to install the gcloud plugin.

** Expected result
The program should run to completion.

rtx doctor output

✗ ~/src/github.com/jdx/rtx/target/debug/rtx doctor
rtx version:
  2023.9.0-DEBUG macos-arm64 (f333ac5 2023-09-07)

build:
  Target: aarch64-apple-darwin
  Features: DEFAULT, NATIVE_TLS
  Built: Thu, 07 Sep 2023 13:48:33 +0000
  Rust Version: rustc 1.72.0 (5680fa18f 2023-08-23)
  Profile: debug

shell:
  /opt/homebrew/bin/fish
  fish, version 3.6.1

rtx data directory:
  /Users/xxx/.local/share/rtx

rtx environment variables:
  RTX_SHORTHANDS_FILE=~/.config/rtx/shorthands.toml
  RTX_SHELL=fish
  RTX_ENV=development

settings:
  {"always_keep_download": "false", "always_keep_install": "false", "asdf_compat": "false", "disable_default_shorthands": "false", "disable_tools": "[]", "experimental": "true", "jobs": "4", "legacy_version_file": "true", "legacy_version_file_disable_tools": "[]", "log_level": "INFO", "missing_runtime_behavior": "warn", "plugin_autoupdate_last_check_duration": "10080", "raw": "false", "shorthands_file": "/Users/xxx/.config/rtx/shorthands.toml", "trusted_config_paths": "[]", "verbose": "false", "yes": "false"}

config files:
  /Users/xxx/.config/rtx/config.toml
  /Users/xxx/src/github.com/EarnestResearch/gcp-core-infra-admin/.rtx.toml
  /Users/xxx/src/github.com/EarnestResearch/gcp-core-infra-admin/.rtx.development.toml

plugins:
  bun              (core)
  deno             (core)
  gcloud           https://github.com/jthegedus/asdf-gcloud.git#dc1aa79
  github-cli       https://github.com/bartlomiejdanek/asdf-github-cli.git#e0605b7
  go               (core)
  java             (core)
  jq               https://github.com/lsanwick/asdf-jq.git#f02032e
  just             https://github.com/olofvndrhr/asdf-just.git#93771e1
  node             (core)
  python           (core)
  ruby             (core)
  rust             https://github.com/code-lever/asdf-rust.git#95acf4f
  shellcheck       https://github.com/luizm/asdf-shellcheck.git#b160a52
  shfmt            https://github.com/luizm/asdf-shfmt.git#a42c5ff
  terraform        https://github.com/asdf-community/asdf-hashicorp.git#0346f5d
  terraform-docs   https://github.com/looztra/asdf-terraform-docs.git#acfd01c
  terraform-ls     https://github.com/asdf-community/asdf-hashicorp.git#0346f5d
  tflint           https://github.com/skyzyx/asdf-tflint.git#37e7eb9
  tiny             https://github.com/rtx-plugins/rtx-tiny.git#df03b67

toolset:
  tflint@latest, terraform-docs@latest, terraform-ls@latest, terraform@1.5, just@1, github-cli@2, gcloud@latest, rust@latest```

[fiadliel]

rtx hangs if either the height or width dimension of the terminal is reasonably small. No hang if I expand the terminal to - I don't know - 120x100?

[aswadsun]

I got the same issue when installing go, regadless of the terminal size. In macos monterey., it will cause memory full and system restart. This also happened in linux, but it only killed the terminal, not causing system restart as macos.

[jdx]

might be related to #813 which has something to do with the openssl library

[jdx]

is everyone here on Sonoma?

[fiadliel]

Just Ventura 13.5.2 here.
By the way, I'm not certain that both issues mentioned here are the same. For the problem I saw before opening this issue, I experienced 100% CPU, but not increasing memory usage. I think this was caused by a panic in Drop from indicativ (because it unwraps a poisoned mutex after the first panic), and who knows what can happen with an unwindable panic in optimised code? I'm fairly sure it isn't openssl, because a crash with understandable backtrace occurred with a debug build pointing to a known indicatif issue.

I have separately seen rtx kill my machine, but haven't yet opened an issue for that until I can reproduce. What I remember seeing was what looked like a fork bomb - over 1000 calls to rtx running simultaneously - I think they might have been running list-all. I also had (but no longer have) the shims directory in my default shell path, and some globally installed plugins - both of these seem possibly relevant to rtx recursively calling itself.

[fiadliel]

I hope that this particular crash is fixed when rtx updates to indicatif 0.17.7 - console-rs/indicatif#586 was merged and is in that release.

[jdx]

well I just hit this error locally:

rtx shellcheck@0.9 [ERROR] /Users/jdx/src/rtx/e2e/.rtx/plugins/shellcheck/ ✗ 0s The application panicked (crashed).
Message:  assertion failed: self.orphan_lines_count <= self.lines.len()
Location: /Users/jdx/.cargo/registry/src/index.crates.io-6f17d22bba15001f/indicatif-0.17.7/src/draw_target.rs:501

so it looks like this is related

[jdx]

I'm seeing this error more often but I think it only happens on debug builds for me.

As a workaround try building it with --release