all: umbrella issue for release-blocking fuzzing work

[katiehockman]

This is the tracking issue for fuzzing-related work which must be done before fuzzing can be part of a standard release of the Go toolchain.

The current plan is to merge the fuzzing code into master once the tree opens in the fall of this year (2021).
This would make fuzzing available for Go 1.18.

The following is a list of release-blocking features (in no particular order):

• Merge fuzzing into master, removing the default build tag (gofuzzbeta) currently in the dev.fuzz branch
• (can be during freeze) Documentation describing which types are currently supported for fuzzing
• A decision must be made on whether or not fuzzing will support generics, as adding this later would break backwards compatibility for existing fuzz targets
  • [dev.fuzz] testing: use generics with testing.F #46890
• Minimization of interesting inputs
• A decision must be made for how -cpu is used
  • cmd/go: respect -cpu=n on workers #46629
• A decision must be made on whether other tests run when -fuzz is set
  • [dev.fuzz] cmd/go: don't run the other tests if -fuzz is specified #46222
• A decision must be made on whether types can be converted when using f.Add and f.Fuzz (may decide to make this more strict for now)
  • [dev.fuzz] testing: crash due to incompatible type #45593
• Don't store corpus in memory
  • [dev.fuzz] internal/fuzz: don't store corpus in memory #46669
• Fuzzing works with -race and -msan. It doesn't have to add a lot of additional value, but the go tool shouldn't break.

The following is a list of nice-to-have features (in a rough order):

• Detect timeouts and provide a -timeout flag.
  • cmd/go: add per-test timeouts #48157
• Allow multiple fuzz targets to be run with a single go command
  • cmd/go: allow fuzzing multiple targets per package #46312
• The mutators might be a bit too aggressive, and could be improved (cmd/go: simple off-by-one example takes longer than expected to complete #47090)
• A tool that can convert binary files into the fuzz encoding file format
  • Related to cmd/go: decide where fuzz tooling should live #46311 (comment)
• Implement better deduplication of interesting inputs so the ondisk corpus doesn't get too inflated
• Deduplication of crashes
• Support for -keepfuzzing (This is contingent on deduplication working first)
• Support for fuzzing more builtin types, such as slices and arrays
• Supporting dictionaries
• Support for fuzzing structs
• Prioritization of newly discovered interesting inputs, and perhaps some minimal work done on newly discovered interesting inputs.
• Improved mutator - specifically one that uses other corpus entries while mutating
• Supports a build mode which can generate a compiled binary to be run using libFuzzer

There are other issues with the 'fuzz' label which will also be release blockers once fuzzing is merged. Once the tree is open, the release blocker label will be used at that time.

Which items are in the "release-blocker" list and which are in the "nice-to-have" list may change over time, depending on user feedback, and new items may be added at any time as well.

/cc @golang/fuzzing

[gopherbot]

Change https://golang.org/cl/351118 mentions this issue: cmd/go: disable gofuzzbeta build tag by default

[gopherbot]

Change https://golang.org/cl/353430 mentions this issue: go.dev/_content/blog: update build constraints in fuzz-beta post

[AlekSi]

Fuzzing works with -race and -msan. It doesn't have to add a lot of additional value, but the go tool shouldn't break.

Currently, -race reports a data race in SnapshotCoverage(). Is it a known problem? Should there be a separate ticket for that?

[AlekSi]

(Also, this ticket is a bit confusing. For example, the title says "release-blocking", but the milestone is "Unreleased")

[katiehockman]

I went ahead and updated the milestone. (This is just an umbrella issue used for tracking and external visibility, so any issues/requests should be filed as separate tickets)

@AlekSi could you file a new issue for the -race issue. We need to look into that again, and AFAIK there isn't an open ticket for it. Thanks.

[katiehockman]

@rolandshoemaker tried to reproduce that race issue locally, and wasn't able to. I'm going to go ahead and mark that done in the umbrella issue, and just close this. This issue was really just for tracking anyway.

But @AlekSi please still file your issue if you can reproduce it at tip.

[thepudds]

FWIW, I’ve been using -race and -fuzz together a fair amount for the past several weeks without an issue, so maybe it is related to the exact code under test.

[AlekSi]

#50488

[katiehockman]

Thanks, @AlekSi for filing that issue!