Temper is a framework for modelling concurrency and failure in distributed systems. The name comes from Temporal Fuzzing, a term coined by Rachel Kroll.
Similar to Loom, programs targeting Temper should be able to switch between simulation mode, and calling real APIs in release mode.
It is in early development, and is not yet more than an experiment. It currently features:
- Low level x86/ARM memory models
- Rust/C++ 11 memory model
Planned features:
- MESI protocol simulation to measure cache line contention and false sharing
- Data race detection
- TCP/IP, including congestion, asymmetric net splits, and Byzantine faults
- Disk operations, including fsync and power failure corruption
- SQL transactional isolation
Related Work:
- Madsim, a similar project with an emphasis on networking
- FoundationDB's testing strategy
- TigerBeetle's fault injection testing
- Loom, which exhaustively tests on a single node
- Timecraft, a distributed system testing tool for WebAssembly
- Coyote, a similar project for the .Net runtime
- Antithesis, a deterministic hypervisor that can test arbitrary software
- Turmoil, a testing framework for Tokio that also does network fault injection
- FrostDB's testing strategy, compiling Go to WASM
Reading:
- Files are fraught with peril by Dan Luu
- What's the big deal about Deterministic Simulation Testing? by Phil Eaton
Memlog simulates the Rust memory model (C++ 11 without Consume). Combined with operation reordering in Temper, its goal is full coverage. It contains a series of test cases dervied from Preshing on Programming, C++ Concurrency in Action, the C++ Standard, blog posts and many Stack Overflow questions.
Todo:
- Detect data races in non-atomic datatypes
- Expose API to declare what can be reordered
- MESI protocol simulation
- Locks
- Seeded randomness
- Reentry support for fetch_update
- Support multiple datatypes
Temper contains a low level simulation of x86/ARM memory models. It is intended for experimentation, as the operations cannot be translated to standard Rust calls in release mode.
Todo:
- Non-coherent memory models (Alpha)
- Locks
- CAS
- Platform specific barriers
- Spin forever under contention
- Crate swap mechanism for release
- Sample lock free algorithms, such as a MPMC queue
- Deterministic testing with seeds and reproducibility
- Disk w/ fsync, power failure, corruption
- Sample Disk LSM system
- TCP with net splits, latency and Byzantine faults
- Sample Raft protocol
- Visualisation