[dev.fuzz] proposal: replace -keepfuzzing with -[test.]failfast

[AlekSi]

-keepfuzzing
    Keep running the target if a crasher is found. (default false)
-failfast
    do not start new tests after the first test failure

Is it possible to change -failfast's implicit value from false to true if -fuzz flag is passed? If yes, I think -keepfuzzing flag could be removed as it means virtually the same thing.

[AlekSi]

@gopherbot fuzz

[cherrymui]

cc @katiehockman

[katiehockman]

The problem with using test.failfast is that the default value, false, would be incorrect when fuzzing. If we translate that to fuzzing, a default where "do not start new tests after the first test failure" is false would mean that the fuzzing engine should keep running after the first test failure. That's the opposite of the default behavior of fuzzing today, where it panics and stops after it finds the first crash. There could be an argument to be made that the default behavior should change, in which case it may make sense to combine the flags, but that's a different conversation.

[AlekSi]

The problem with using test.failfast is that the default value, false, would be incorrect when fuzzing.

Yes, that why I asked if we can change "implicit value from false to true if -fuzz flag is passed"? We do change the default value of -timeout flag at #44483 (comment), so maybe we should do the same there?

[katiehockman]

Yes, that why I asked if we can change "implicit value from false to true if -fuzz flag is passed"?

We could, but keep in mind that the current behavior of go test -fuzz is to run all the other tests in that package, and all of the seed corpus, and if all of those succeed, start to fuzz. We could consider changing that, but let's assume for now that this is the behavior that we want. I could see a user wanting to run go test -fuzz=FuzzFoo -failfast=false -keepfuzzing=false, which would mean "Run the unit tests and report all failures that occurred. If those succeeded, then start fuzzing, and stop when you find the first crasher." That wouldn't be possible if we combine the flags into one.

Consider the default behavior:
Someone runs go test -fuzz=FuzzFoo, which defaults to -failfast=true since -fuzz is set. That means that if any of the unit tests fail, then it will fail fast and won't run the rest of the unit tests. That would surprise me, personally. As a user, I would want all of the unit tests and the seed corpus to be run in the exact same way with go test -fuzz as it would with go test by default, with the only difference being that the fuzzing engine will run if all of the tests pass.

[AlekSi]

Thank you for the explanation; I understand it now and agree with you. I am closing the issue.

As a side note, we did a workshop about fuzzing using dev.fuzz on GopherCon Russia last Sunday (in Russian), and it went great! Looking forward to dev.fuzz being shipped.