cmd/go: allow fuzzing multiple targets per package

[jayconrod]

Currently, go test only allows fuzzing one target per package. So if package example.com/a has targets Fuzz1 and Fuzz2, this command fails:

$ go test -fuzz=. example.com/a
ok  	example.com/a	0.103s [will not fuzz, -fuzz matches more than one target]

(Actually, it still exits zero without fuzzing, which is probably bad, too).

If multiple packages have individual fuzz targets, that's okay though:

$ go test -fuzz=. example.com/a example.com/b

This seems a little strange. We should decide to either allow one fuzzing target globally across all matched packages on the go test command line, or allow multiple targets per package and fuzz them all in parallel.

[jayconrod]

For reference, the -bench flag can match multiple benchmarks in multiple packages. We should probably do that.

I think go test -bench=. ./... will run benchmarks sequentially to avoid contention. We could either do that with -fuzz (with -fuzztime saying how long to run), or we could coordinate multiple fuzz targets concurrently.

[katiehockman]

We should decide to either allow one fuzzing target globally across all matched packages on the go test command line, or allow multiple targets per package and fuzz them all in parallel.

My preference would be to allow multiple targets per package to be fuzzed in parallel
Edit (since I think that was unclear): What I'd like to support in the future is running several targets, perhaps even across several packages, in some kind of rotation. So that might look like running target (1) for one second, then (2) for one second, etc etc.

I think go test -bench=. ./... will run benchmarks sequentially to avoid contention. We could either do that with -fuzz (with -fuzztime saying how long to run), or we could coordinate multiple fuzz targets concurrently.

I feel like running multiple fuzz targets concurrently is just asking for trouble (e.g. races). Though I guess if each is in it's own process, maybe the risk of that isn't actually that high. We also have to decide what to do if someone wants to fuzz 10 targets at once but only use 3 processes. At that point we'll have to run them sequentially/in a round anyway.

I think it will be simpler and equally efficient to just run each target for some period of time in a round. For example, we run each target for 3 seconds before moving to the next, and so on, until we're back at the first target, then repeat. I don't see a compelling reason why the "time to run each target before moving to the next" needs to be user-configurable, but LMK if you can think of a reason.

[gopherbot]

Change https://golang.org/cl/350156 mentions this issue: [dev.fuzz] cmd/go: in 'go test' don't allow multiple packages with -fuzz

[nightlyone]

OK that is getting weird now: Are you now supposed to iterate over the list of fuzzable packages yourself?

Useful CI systems usually run go ACTION ./... for any major action the go command offers.

If that is not possible, could you at least provide some way to create the full list of fuzz parallel jobs with the sequential lists of each fuzz function? That together with a max fuzz time would allow CI systems to express the resource budgets itself.

[katiehockman]

@nightlyone

Are you now supposed to iterate over the list of fuzzable packages yourself?

For now. This isn't necessarily the long term solution for fuzzing. It's just a safety measure that's in place now until we can build this out effectively.

If that is not possible, could you at least provide some way to create the full list of fuzz parallel jobs with the sequential lists of each fuzz function?

That might be possible. We'll take a look and see if we can provide something like this before 1.18.
An initial idea to get us started: One way to do this would be to write a little tool which matches on FuzzX(*testing.F) functions in *_test.go files, and spits out a script which, if run, would fuzz runs each target in a round. Maybe each target is fuzzed for at least 10 seconds as a default, but this could be configurable. We could do this without any code changes to the fuzzing engine or go command. The concern is that we need to make sure the default is long enough. If, for example, the default is 1 second, it might take approximately that long to start everything up and run the warm-up for initial coverage. And since the go command process would completely shut down before starting the next target, we would need to do this warm-up every time. We could do this outside the release cycle (for folks who want it, this can just go in x/tools or something).

[ethanmdavidson]

Here's the quick-n-dirty bash solution:

#!/bin/bash

set -e

fuzzTime=${1:-10}

files=$(grep -r --include='**_test.go' --files-with-matches 'func Fuzz' .)

for file in ${files}
do
	funcs=$(grep -oP 'func \K(Fuzz\w*)' $file)
	for func in ${funcs}
	do
		echo "Fuzzing $func in $file"
		parentDir=$(dirname $file)
		go test $parentDir -run=$func -fuzz=$func -fuzztime=${fuzzTime}s
	done
done

[benjivesterby]

Here's an updated script that works a little more broadly

#!/bin/bash

set -e

fuzzTime=${1:-10}

files=$(grep -r --include='**_test.go' --files-with-matches 'func Fuzz' .)

for file in ${files}
do
	funcs=$(grep -o 'func Fuzz\w*' $file | sed 's/func //')
	for func in ${funcs}
	do
		echo "Fuzzing $func in $file"
		parentDir=$(dirname $file)
		go test $parentDir -run=$func -fuzz=$func -fuzztime=${fuzzTime}s
	done
done

[javking07]

Here's the quick-n-dirty bash solution:

#!/bin/bash

set -e

fuzzTime=${1:-10}

files=$(grep -r --include='**_test.go' --files-with-matches 'func Fuzz' .)

for file in ${files}
do
	funcs=$(grep -oP 'func \K(Fuzz\w*)' $file)
	for func in ${funcs}
	do
		echo "Fuzzing $func in $file"
		parentDir=$(dirname $file)
		go test $parentDir -run=$func -fuzz=$func -fuzztime=${fuzzTime}s
	done
done

@benjivesterby does grep actually support a -P flag? ran this on macos and run into:

grep: invalid option -- P
usage: grep [-abcdDEFGHhIiJLlMmnOopqRSsUVvwXxZz] [-A num] [-B num] [-C[num]]
        [-e pattern] [-f file] [--binary-files=value] [--color=when]
        [--context[=num]] [--directories=action] [--label] [--line-buffered]
        [--null] [pattern] [file ...]

[acaird]

this is what i use on MacOS

#!/bin/bash

fuzzTime=${1:-1h}

while true; do
        files=$(grep -r --include='**_test.go' --files-with-matches 'func Fuzz' .)
        for file in ${files}; do
                funcs=$(grep '^func Fuzz' "$file" | sed s/func\ // | sed 's/(.*$//')

                for func in ${funcs}; do
                        echo "Fuzzing $func in $file"
                        parentDir=$(dirname "$file")
                        go test "$parentDir" -run="$func" -fuzz="$func" -fuzztime="${fuzzTime}"
                done
        done
done