[dev.fuzz] cmd/go: don't run the other tests if -fuzz is specified

[katiehockman]

The current behavior of go test when -fuzz is specified is to run all of the unit tests, examples, and benchmarks first, then start fuzzing only if all the other tests succeed. The reason behind this is that if a new bug has been introduced and an existing test would already catch this issue, then attempting to discover and report a "new" crasher could end up being redundant to the existing tests. Thus, fuzzing only begins once the go command has exhausted all other ways of finding a bug.

However, if there are a lot of other tests, this can delay the start of fuzzing in a way that may surprise developers. When I ran fuzz targets locally, I often found myself wanting to execute something like
go test -run=FuzzFoo -fuzz=FuzzFoo

We may want to consider changing the default behavior based on user experience feedback.

[iand]

I would expect fuzzing to work like benchmarks, i.e. run the tests first unless I disable them (-run=^$ or similar)

[rsc]

Strong agree with @iand.

Fuzzing and benchmarks have in common that you end up making changes that may actually cause unintended breakages. You want to find that out sooner rather than later (your results stop being valid at the point where you break the code), so the tests should be run by default.

This is the right behavior even if user feedback complains about it. You are not hearing from the users who are helped by this, who have not lost hours and hours to fuzzing (or benchmarking) broken code.

[rsc]

Not sure if @jayconrod filed another issue, but it would make sense to check whether a fuzz test matches the -fuzz pattern and if so not run just that one test function as an ordinary test case, because that duplicates work and also because the fuzzing harness can do a better job with infinite loops.

[katiehockman]

Skipping the fuzz target as an ordinary test case seems like a reasonable option. I didn't see a bug filed for this, so I went ahead and filed one: #48296

I am considering this new issue lower priority since it's more of an optimization than a bug, so not a release-blocker. The seed corpus will get run either way, it's just that we can likely make it so that they are only run once (by the fuzz workers) instead of as a typical unit test. But we'll try to get it done before the release. Comment on that bug if you disagree, though.

I'll close this original bug since it sounds like we're all in agreement that we shouldn't skip all other tests when fuzzing.