expose random seed to the test runner

[andrewrk]

In #14940, the Zig compiler passes a 32-bit integer --seed parameter to the build runner for shuffling dependency traversal order. If something goes wrong, the user can repeat the same operation with the same seed because the seed will be printed on the command line.

This same concept can be useful for unit tests which want to use randomness but also need reproducibility.

This proposal is to expose the seed to std.testing so that unit tests can obtain it and use it.

This would also provide a way to resolve one area of confusion for users which is the fact that unit tests get cached. Some users are confused why unit tests would be cached. Explicitly having this seed would make everything make sense. If you give the same seed, then the test runs are considered cached. If you give a different seed, it's a new run.

Example code:

test "example" {
    var rng = std.DefaultPrng.init(std.testing.random_seed);
    // use rng to influence the unit test
}

If this is implemented then it will be important for the seed to be displayed if the tests fail. This is already the case, as shown here:

[nix-shell:~/dev/zig/build-release]$ stage3/bin/zig test test3.zig 
Test [1/1] test.example... FAIL (TestUnexpectedResult)
/home/andy/dev/zig/lib/std/testing.zig:527:14: 0x22383f in expect (test)
    if (!ok) return error.TestUnexpectedResult;
             ^
/home/andy/dev/zig/lib/test3.zig:5:5: 0x223955 in test.example (test)
0 passed; 0 skipped; 1 failed.
error: the following test command failed with exit code 1:
/home/andy/dev/zig/zig-cache/o/1a7789de4ed95c05d2d1ac588f0e29ca/test

This proposal would mean that last line would include, for example, --seed 0xdeadbeef .

[tauoverpi]

Does this affect all tests or would it only affect tests which touch the seed? I would expect tests which don't make use of the seed to remain cached unless the user forces a re-run of everything.

[Rexicon226]

I think this is a really good point. It makes sense to maximize the amount of caching that will still be happening. Would it be possible to still cache the "seeded" tests, and only use the cache if the seed is the same when re-run?

[nektro]

if the seed is a decl/build_option then zig's lazy analysis then this will ensure the caching is preserved as only the tests using the seed will touch it and be affected by its changes

[matklad]

yes yes please I love this soooo much, we need this a tone at TigerBeetle!

Sorry for being overly enthusiastic here, it's just such a brilliant solution to the "reproduciblity vs true randomness" dilemma in randomized tests, beautiful!

If this is implemented then it will be important for the seed to be displayed if the tests fail. This is already the case, as shown here:

I think, with the server setup, we can display the seed even when the tests crashes due to assert? That would be very helpful, as a lot of failures are not "clean", and inability to print the seed during panic makes it hard to implement this feature today "in userspace".

[andrewrk]

Perhaps the seed could be provided in the build summary for test steps? Would that address the concern? If not could you provide an example shell session where the seed would be problematically missing?

[matklad]

So what we do today is

test "k_way_merge: fuzz" {
    const seed = std.crypto.random.int(u64);
    errdefer std.debug.print("\nTEST FAILED: seed = {}\n", .{seed});

    var prng = std.rand.DefaultPrng.init(seed);
    const random = prng.random();

    // Body of the test //
}

This works if the body of the tests looks like

    try std.testing.expectEqual(random.int(u8), 0);

This doesn't work if your test looks rather like

    std.debug.assert(random.int(u8) == 0);

Here's the printout for that case today:

run test: error: thread 20615 panic: reached unreachable code
/home/matklad/p/tb/work/zig/lib/std/debug.zig:343:14: 0x227ecc in assert (test)
    if (!ok) unreachable; // assertion failure
             ^
/home/matklad/p/tb/work/src/lsm/k_way_merge.zig:470:21: 0x22f22d in test.k_way_merge: fuzz (test)
    std.debug.assert(random.int(u8) == 0);
                    ^
/home/matklad/p/tb/work/zig/lib/test_runner.zig:100:29: 0x22d829 in mainServer (test)
                test_fn.func() catch |err| switch (err) {
                            ^
/home/matklad/p/tb/work/zig/lib/test_runner.zig:34:26: 0x22775a in main (test)
        return mainServer() catch @panic("internal test runner failure");
                         ^
/home/matklad/p/tb/work/zig/lib/std/start.zig:564:22: 0x2270c9 in main (test)
            root.main();
                     ^
???:?:?: 0x7f5dc0b76acd in ??? (libc.so.6)
Unwind information for `libc.so.6:0x7f5dc0b76acd` was not available, trace may be incomplete


run test: error: while executing test 'test.k_way_merge: fuzz', the following command terminated with signal 6 (expected exited with code 0):
/home/matklad/p/tb/work/zig-cache/o/485906540d0b75cec548d0afb33c79b5/test --listen=- 
Build Summary: 6/8 steps succeeded; 1 failed; 2/2 tests passed (disable with --summary none)
test transitive failure
└─ run test failure
error: the following build command failed with exit code 1:
/home/matklad/p/tb/work/zig-cache/o/d668bfb7357d37d5edbea66cab8648cc/build /home/matklad/p/tb/work/zig/zig /home/matklad/p/tb/work /home/matklad/p/tb/work/zig-cache /home/matklad/.cache/zig test -Dtest-filter=k_way_merge: fuzz

And most of the tests where we use randomization are of the second kind, where we don't check for specific expected values, but rather rely on internal assertions to verify invariant. (that is, the assert is not in the test directly, but rather deep in the call-stack triggered by the test).

There are some unsatisfactory ways we could fix this today:

• just always print the seed (but that goes against the guideline that the output should be empty, unless there's a real reason to print smothing)
• use something like libc at exit, or some global variable and our custom panic handler function, to print the seed "at the last minute". This feels hacky and ineligant, and might not work, as the process state by the time of the crash can be arbitrary messed up.

The fundamental problem here is that the seed is created inside the process, and, if that process is itself dying, you can't reliably print the seed.

That's why I really love this solution, where the seed is explicitly passed in by the parent process --- the parent process should have the ability to print the seed even if the child process dies abruptly (or, indeed, if it gets into an infinite loop).

Perhaps the seed could be provided in the build summary for test steps?

Ideally, I'd love to see the seed only when there's a test failure. It's too easy to "just print more helpful info", but this dynamics leads to software which prints pages of output by default, making the output useless due to bad singnal/noise ratio.

[andrewrk]

Thanks for the example. I think this would already be solved by the original proposal. Example:

const std = @import("std");
const assert = std.debug.assert;

test "example" {
    assert(false);
}

$ zig test test.zig 
Test [1/1] test.example... thread 4057934 panic: reached unreachable code
/home/andy/Downloads/zig/lib/std/debug.zig:342:14: 0x2237f2 in assert (test)
    if (!ok) unreachable; // assertion failure
             ^
/home/andy/Downloads/zig/lib/test.zig:5:11: 0x2237aa in test.example (test)
/home/andy/Downloads/zig/lib/test_runner.zig:181:28: 0x22c35d in mainTerminal (test)
        } else test_fn.func();
                           ^
/home/andy/Downloads/zig/lib/test_runner.zig:36:28: 0x22462a in main (test)
        return mainTerminal();
                           ^
/home/andy/Downloads/zig/lib/std/start.zig:573:22: 0x223cbc in posixCallMainAndExit (test)
            root.main();
                     ^
/home/andy/Downloads/zig/lib/std/start.zig:251:5: 0x223811 in _start (test)
    asm volatile (switch (native_arch) {
    ^
???:?:?: 0x0 in ??? (???)
error: the following test command crashed:
/home/andy/Downloads/zig/zig-cache/o/28ff5b8969535a9e5972718ca920cf01/test --seed 0xdeadbeef

The only edit I made to this copy-pasted output here is adding --seed 0xdeadbeef to the end, to demonstrate what the parent process would start doing.

In your example, it would show up here:

/home/matklad/p/tb/work/zig-cache/o/485906540d0b75cec548d0afb33c79b5/test --seed 0xdeadbeef --listen=-