integrate fuzz testing into the build system #20773

andrewrk · 2024-07-24T07:15:55Z

Adds a --fuzz CLI option to the build runner. When this is used it rebuilds any unit test binaries which contained at least one fuzz test with -ffuzz and then tells it to start fuzzing, which does in-process fuzzing.

Adds std.testing.fuzzInput, which is how unit tests mark themselves as fuzz tests.

This contains only a rudimentary implementation of fuzzer logic, really just some early, early experiments, but already it makes this test case fail in 65 milliseconds on my machine:

test "fuzz example" {
    const input_bytes = std.testing.fuzzInput(.{});
    try std.testing.expect(!std.mem.eql(u8, "canyoufindme", input_bytes));
}

andy@bark ~/t/abc [1]> ~/dev/zig/build-release/stage4/bin/zig build test --fuzz --debug-rt
test
└─ run test failure
/home/andy/dev/zig/lib/std/testing.zig:546:14: 0x1183a59 in expect (test)
    if (!ok) return error.TestUnexpectedResult;
             ^
/home/andy/tmp/abc/src/main.zig:28:5: 0x1183b21 in test.fuzz example (test)
    try std.testing.expect(!std.mem.eql(u8, "canyoufindme", input_bytes));
    ^
failed with error.TestUnexpectedResult
error: the following command exited with error code 1:
/home/andy/tmp/abc/.zig-cache/o/15697f80a9140388803537517083788c/test --seed=0x93a767fd --listen=- 
error: all fuzz workers crashed
error: the following build command failed with exit code 1:
/home/andy/tmp/abc/.zig-cache/o/12ab17aae32a6d5d6c9fba7b28ba3167/build /home/andy/dev/zig/build-release/stage4/bin/zig /home/andy/dev/zig/lib /home/andy/tmp/abc /home/andy/tmp/abc/.zig-cache /home/andy/.cache/zig --seed 0x93a767fd -Zb1f080d4c8ac26d5 test --fuzz --debug-rt

asciinema demo

Closes #20702.

Follow-Up Tasks

false positive cache hit on compiler_rt and libfuzzer when used via the build system #20782
support code coverage when testing #352
Proposal: Add a way for a test to expect a panic #1356
Consider running tests in parallel by deafult #15953
unit tests: enforce time limits #19821
write fuzz inputs to a shared memory region before running a task #20803
enhance the fuzzing algorithm to be competitive with other mainstream fuzzers #20804
add a UI to fuzzing to report progress, code coverage, interesting inputs, and other stats #20812
std.testing.fuzzInput: introduce a corpus option #20814
std.testing.fuzzInput: ability to choose whether timing out is considered a fail or a pass #20815
std.testing.fuzzInput: introduce a length range option #20816
unique failure detection
corpus minimization mode
more flags to the build runner such as timed fuzzing
investigate adding sometimes assertions, perhaps with scopes
investigate coordination between multiple zig build --fuzz instances running on different machines
integrate --fuzz with --watch

for the -fno-fuzz case. The other case will take more work in libfuzzer.

For now this returns a dummy fuzz input.

This flag makes the build runner rebuild unit tests after the pipeline finishes, if it finds any unit tests. I did not make this integrate with file system watching yet. The test runner is updated to detect which tests are fuzz tests. Run step is updated to track which test indexes are fuzz tests.

This way they can be smoke tested.

The flag makes compiler_rt and libfuzzer be in debug mode. Also: * fuzzer: override debug logs and disable debug logs for frequently called functions * std.Build.Fuzz: fix bug of rerunning the old unit test binary * report errors from rebuilding the unit tests better * link.Elf: additionally add tsan lib and fuzzer lib to the hash

just some experimentation. I didn't expect this to be effective so quickly but it already can find a comparison made with mem.eql

seems to provide better scoring

rohlem · 2024-07-27T18:38:59Z

investigate adding sometimes assertions, perhaps with scopes

In case these get added I want to propose the name assertReachable.

andrewrk added zig build system std.Build, the build runner, `zig build` subcommand, package management release notes This PR should be mentioned in the release notes. labels Jul 24, 2024

andrewrk force-pushed the fuzz branch 3 times, most recently from 726cec6 to 6c68370 Compare July 25, 2024 22:17

andrewrk added 12 commits July 25, 2024 18:52

implement std.testing.fuzzInput

5058beb

for the -fno-fuzz case. The other case will take more work in libfuzzer.

llvm: always include instrumentation function attributes

3256df2

implement std.testing.fuzzInput

6f37678

For now this returns a dummy fuzz input.

build runner: extract logic to std.Build.Fuzz

711ed56

build runner sends a start_fuzzing message to test runner

bce3b1e

test runner: handle start_fuzzing message

7366b4b

std.Build.Fuzz: fix progress node hierarchy

b501adc

test runner: always report fuzz tests

90dfd86

This way they can be smoke tested.

fuzzer: basic implementation

6a63372

just some experimentation. I didn't expect this to be effective so quickly but it already can find a comparison made with mem.eql

fuzzer: use the cmp values

688c2df

seems to provide better scoring

andrewrk force-pushed the fuzz branch from 6c68370 to 688c2df Compare July 26, 2024 01:52

andrewrk merged commit afddfe2 into master Jul 26, 2024
0 of 10 checks passed

andrewrk deleted the fuzz branch July 26, 2024 01:52

andrewrk added the fuzzing label Jul 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

integrate fuzz testing into the build system #20773

integrate fuzz testing into the build system #20773

andrewrk commented Jul 24, 2024 •

edited

Loading

rohlem commented Jul 27, 2024

integrate fuzz testing into the build system #20773

integrate fuzz testing into the build system #20773

Conversation

andrewrk commented Jul 24, 2024 • edited Loading

Follow-Up Tasks

rohlem commented Jul 27, 2024

andrewrk commented Jul 24, 2024 •

edited

Loading