Cache::get method sometimes panics due to an integer overflow in quanta::instant::Instant::now

[tatsuya6502]

This issue was reported by @BrynCooke on May 10, 2022 UTC via #113 (comment)

The number of times needed to hit this is very low. Sometimes it happens within 10 invocations, sometimes 100.

---

I have been hitting this issue and am able to reproduce by running an integration test in a loop.

Looks like it is possibly related to this: metrics-rs/quanta#61

The stacktrace is as follows:

2022-05-10T11:16:52.903652Z ERROR integration_tests: panicked at 'attempt to add with overflow', /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/num/mod.rs:834:5


1: std::panicking::rust_panic_with_hook
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:702:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:586:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/sys_common/backtrace.rs:138:18
   4: rust_begin_unwind
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:584:5
   5: core::panicking::panic_fmt
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:143:14
   6: core::panicking::panic
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:48:5
   7: core::num::<impl u64>::next_power_of_two
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/num/uint_macros.rs:2188:13
   8: quanta::Calibration::adjust_cal_ratio
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/quanta-0.9.3/src/lib.rs:272:25
   9: quanta::Calibration::calibrate
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/quanta-0.9.3/src/lib.rs:226:13
  10: quanta::Clock::new::{{closure}}
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/quanta-0.9.3/src/lib.rs:307:17
  11: once_cell::sync::OnceCell<T>::get_or_init::{{closure}}
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/once_cell-1.9.0/src/lib.rs:974:57
  12: once_cell::imp::OnceCell<T>::initialize::{{closure}}
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/once_cell-1.9.0/src/imp_std.rs:95:19
  13: once_cell::imp::initialize_inner
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/once_cell-1.9.0/src/imp_std.rs:171:31
  14: once_cell::imp::OnceCell<T>::initialize
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/once_cell-1.9.0/src/imp_std.rs:93:9
  15: once_cell::sync::OnceCell<T>::get_or_try_init
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/once_cell-1.9.0/src/lib.rs:1014:13
  16: once_cell::sync::OnceCell<T>::get_or_init
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/once_cell-1.9.0/src/lib.rs:974:19
  17: quanta::Clock::new
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/quanta-0.9.3/src/lib.rs:305:31
  18: core::ops::function::FnOnce::call_once
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/ops/function.rs:227:5
  19: once_cell::sync::OnceCell<T>::get_or_init::{{closure}}
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/once_cell-1.9.0/src/lib.rs:974:57
  20: once_cell::imp::OnceCell<T>::initialize::{{closure}}
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/once_cell-1.9.0/src/imp_std.rs:95:19
  21: once_cell::imp::initialize_inner
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/once_cell-1.9.0/src/imp_std.rs:171:31
  22: once_cell::imp::OnceCell<T>::initialize
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/once_cell-1.9.0/src/imp_std.rs:93:9
  23: once_cell::sync::OnceCell<T>::get_or_try_init
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/once_cell-1.9.0/src/lib.rs:1014:13
  24: once_cell::sync::OnceCell<T>::get_or_init
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/once_cell-1.9.0/src/lib.rs:974:19
  25: quanta::get_now
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/quanta-0.9.3/src/lib.rs:532:9
  26: quanta::instant::Instant::now
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/quanta-0.9.3/src/instant.rs:25:9
  27: moka::common::time::Instant::now
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/moka-0.8.2/src/common/time.rs:24:17
  28: moka::sync::base_cache::Inner<K,V,S>::current_time_from_expiration_clock
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/moka-0.8.2/src/sync/base_cache.rs:760:13
  29: moka::sync::base_cache::BaseCache<K,V,S>::get_with_hash::{{closure}}
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/moka-0.8.2/src/sync/base_cache.rs:170:23
  30: moka::cht::map::bucket_array_ref::BucketArrayRef<K,V,S>::get_key_value_and_then
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/moka-0.8.2/src/cht/map/bucket_array_ref.rs:48:30
  31: moka::cht::segment::HashMap<K,V,S>::get_key_value_and_then
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/moka-0.8.2/src/cht/segment.rs:306:9
  32: moka::sync::base_cache::Inner<K,V,S>::get_key_value_and_then
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/moka-0.8.2/src/sync/base_cache.rs:630:9
  33: moka::sync::base_cache::BaseCache<K,V,S>::get_with_hash
             at /home/bryn/.asdf/installs/rust/1.60.0/registry/src/github.com-1ecc6299db9ec823/moka-0.8.2/src/sync/base_cache.rs:167:27
  34: moka::sync::cache::Cache<K,V,S>::get

The number of times needed to hit this is very low. Sometimes it happens within 10 invocations, sometimes 100.

[tatsuya6502]

@BrynCooke

Looks like it is possibly related to this: metrics-rs/quanta#61

Yeah. It looks like so. From your stack trace, I see quanta::instant::Instant::now function is causing the panic.

metrics-rs/quanta#61 (comment)

My Lenovo laptop is AMD based, and it happens pretty quickly.

It also occurred once on a CircleCI run.

My colleagues, who have M1 and Intel CPUs have not been able to reproduce.

As a workaround, I am going to add a Cargo feature to Moka to opt-out Quanta.

Also, when I have more time, I will try to reproduce the issue by using AMD based instance at Amazon EC2.

[tatsuya6502]

As a workaround, I am going to add a Cargo feature to Moka to opt-out Quanta.

Memo: I could reuse the codes here: #39

[tatsuya6502]

As a workaround, I am going to add a Cargo feature to Moka to opt-out Quanta.

Working on this workaround via #121. Not quite finished yet.

[tatsuya6502]

Hi @BrynCooke,

I have published Moka v0.8.3 to crates.io. Now you can opt out Quanta crate by adding default-features = false to the Moka dependency in your Cargo.toml. For more details, please see this section of the README.

Hope this helps.

I will keep this GH Issue open until Quanta addresses the issue.

[BrynCooke]

@tatsuya6502 Thanks for sorting this out so quickly!

[tatsuya6502]

Hi @BrynCooke — I have published Moka v0.8.4 to crates.io. It depends on the latest Quanta v0.10.0 that has the fix for the panic issue. Please try Moka v0.8.4 when you have time. Thanks!

[BrynCooke]

I had a go and can no longer reproduce the issue. Thanks @tatsuya6502!

[tobz]

Glad to hear the issue is fixed. 👍🏻