Commits on Sep 30, 2024

1. Port sort-research-rs test suite Rust stdlib tests …

   This commit is a followup to rust-lang#124032. It
   replaces the tests that test the various sort functions in the standard library
   with a test-suite developed as part of
   https://github.com/Voultapher/sort-research-rs. The current tests suffer a
   couple of problems:
   
   - They don't cover important real world patterns that the implementations take
     advantage of and execute special code for.
   - The input lengths tested miss out on code paths. For example, important safety
     property tests never reach the quicksort part of the implementation.
   - The miri side is often limited to `len <= 20` which means it very thoroughly
     tests the insertion sort, which accounts for 19 out of 1.5k LoC.
   - They are split into to core and alloc, causing code duplication and uneven
     coverage.
   - The randomness is not repeatable, as it
     relies on `std::hash::RandomState::new().build_hasher()`.
   
   Most of these issues existed before
   rust-lang#124032, but they are intensified by it.
   One thing that is new and requires additional testing, is that the new sort
   implementations specialize based on type properties. For example `Freeze` and
   non `Freeze` execute different code paths.
   
   Effectively there are three dimensions that matter:
   
   - Input type
   - Input length
   - Input pattern
   
   The ported test-suite tests various properties along all three dimensions,
   greatly improving test coverage. It side-steps the miri issue by preferring
   sampled approaches. For example the test that checks if after a panic the set of
   elements is still the original one, doesn't do so for every single possible
   panic opportunity but rather it picks one at random, and performs this test
   across a range of input length, which varies the panic point across them. This
   allows regular execution to easily test inputs of length 10k, and miri execution
   up to 100 which covers significantly more code. The randomness used is tied to a
   fixed - but random per process execution - seed. This allows for fully
   repeatable tests and fuzzer like exploration across multiple runs.
   
   Structure wise, the tests are previously found in the core integration tests for
   `sort_unstable` and alloc unit tests for `sort`. The new test-suite was
   developed to be a purely black-box approach, which makes integration testing the
   better place, because it can't accidentally rely on internal access. Because
   unwinding support is required the tests can't be in core, even if the
   implementation is, so they are now part of the alloc integration tests. Are
   there architectures that can only build and test core and not alloc? If so, do
   such platforms require sort testing? For what it's worth the current
   implementation state passes miri `--target mips64-unknown-linux-gnuabi64` which
   is big endian.
   
   The test-suite also contains tests for properties that were and are given by the
   current and previous implementations, and likely relied upon by users but
   weren't tested. For example `self_cmp` tests that the two parameters `a` and `b`
   passed into the comparison function are never references to the same object,
   which if the user is sorting for example a `&mut [Mutex<i32>]` could lead to a
   deadlock.
   
   Instead of using the hashed caller location as rand seed, it uses seconds since
   unix epoch / 10, which given timestamps in the CI should be reasonably easy to
   reproduce, but also allows fuzzer like space exploration.

   

   Voultapher committed Sep 30, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 71bb0e7
   • Browse repository at this point

   Copy the full SHA

   71bb0e7 View commit details

   Browse the repository at this point in the history