Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libFuzzer support (WIP) #217

Merged
merged 1 commit into from
Mar 12, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 96 additions & 8 deletions go-fuzz-build/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,15 +27,21 @@ import (
)

var (
flagTag = flag.String("tags", "", "a space-separated list of build tags to consider satisfied during the build")
flagOut = flag.String("o", "", "output file")
flagFunc = flag.String("func", "Fuzz", "entry function")
flagWork = flag.Bool("work", false, "don't remove working directory")
flagCPU = flag.Bool("cpuprofile", false, "generate cpu profile in cpu.pprof")
flagTag = flag.String("tags", "", "a space-separated list of build tags to consider satisfied during the build")
flagOut = flag.String("o", "", "output file")
flagFunc = flag.String("func", "Fuzz", "entry function")
flagWork = flag.Bool("work", false, "don't remove working directory")
flagCPU = flag.Bool("cpuprofile", false, "generate cpu profile in cpu.pprof")
flagLibFuzzer = flag.Bool("libfuzzer", false, "output static archive for use with libFuzzer")
)

func makeTags() string {
tags := "gofuzz"

if *flagLibFuzzer {
tags += " " + "gofuzz_libfuzzer"
}

if len(*flagTag) > 0 {
tags += " " + *flagTag
}
Expand Down Expand Up @@ -84,6 +90,21 @@ func main() {
// See also https://golang.org/issue/29824.
lits := c.gatherLiterals()
var blocks, sonar []CoverBlock

if *flagLibFuzzer {
archive := c.buildInstrumentedBinary(&blocks, nil)

if *flagOut == "" {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about we unify the decision about what to use for *flagOut. So above:

if *flagOut == "" {
  suffix := ".zip"
  if *flagLibFuzzer {
    suffix = ".a"
  }
  *flagOut = c.pkgs[0].Name+"-fuzz"+suffix
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then if/when we add the Fuzz method name as context, we won't have to add it in three places. (We already need to add it in go-fuzz-build and go-fuzz.)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since libFuzzer doesn't use the literals, let's move this whole thing up higher above the giant comment and make it standalone. Something like:

if *flagLibFuzzer {
  var blocks []CoverBlock
  archive := ...
  os.Rename ...
  return
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the literal finder extract hardcoded strings from the source files?
That might actually be very useful to create a dictionary file from, that libFuzzer can consume via the -dict= parameter. But I'll have to test this.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it does.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also do the last step and run:

clang target.a -fsanitize=fuzzer
``
? This gives me a working fuzzer which is nice.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is interesting but won't work with oss-fuzz which requires that you link against -lFuzzingEngine.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, let's leave it as is for now.
We can get back to this if/when somebody will have interest in using it. We could build both, or have an additional flag.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think we should do my first suggestion in this thread, about adjusting *flagOut only once. But we can merge without that and I can do it as a follow-up.

*flagOut = c.pkgs[0].Name + ".a"
}

err := os.Rename(archive, *flagOut)
if err != nil {
c.failf("failed to rename file: %v", err)
}
return
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks unformatted, please run go fmt ./...

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

}

coverBin := c.buildInstrumentedBinary(&blocks, nil)
sonarBin := c.buildInstrumentedBinary(nil, &sonar)
metaData := c.createMeta(lits, blocks, sonar)
Expand Down Expand Up @@ -257,6 +278,9 @@ func (c *Context) populateWorkdir() {

// TODO: See if we can avoid making toolchain copies,
// using some combination of env vars and toolexec.
if *flagLibFuzzer {
c.copyDir(filepath.Join(c.GOROOT, "src", "runtime", "cgo"), filepath.Join(c.workdir, "goroot", "src", "runtime", "cgo"))
}
c.copyDir(filepath.Join(c.GOROOT, "pkg", "tool"), filepath.Join(c.workdir, "goroot", "pkg", "tool"))
if _, err := os.Stat(filepath.Join(c.GOROOT, "pkg", "include")); err == nil {
c.copyDir(filepath.Join(c.GOROOT, "pkg", "include"), filepath.Join(c.workdir, "goroot", "pkg", "include"))
Expand Down Expand Up @@ -298,14 +322,27 @@ func (c *Context) createMeta(lits map[Literal]struct{}, blocks []CoverBlock, son
return f
}

func extraBuildFlags() string {
if *flagLibFuzzer {
return "-buildmode=c-archive"
}
return ""
}

func (c *Context) buildInstrumentedBinary(blocks *[]CoverBlock, sonar *[]CoverBlock) string {
c.instrumentPackages(blocks, sonar)
mainPkg := c.createFuzzMain(c.pkgpath)

outf := c.tempFile()
os.Remove(outf)
outf += ".exe"
cmd := exec.Command("go", "build", "-tags", makeTags(), "-o", outf, mainPkg)

if !*flagLibFuzzer {
outf += ".exe"
} else {
outf += ".a"
}

cmd := exec.Command("go", "build", "-tags", makeTags(), extraBuildFlags(), "-o", outf, mainPkg)
cmd.Env = append(os.Environ(),
"GOROOT="+filepath.Join(c.workdir, "goroot"),
"GOPATH="+filepath.Join(c.workdir, "gopath"),
Expand Down Expand Up @@ -384,11 +421,19 @@ func (c *Context) copyFuzzDep() {
}
}

func funcMain() string {
if !*flagLibFuzzer {
return mainSrc
}

return mainSrcLibFuzzer
}

func (c *Context) createFuzzMain(pkg string) string {
mainPkg := filepath.Join(pkg, "go.fuzz.main")
path := filepath.Join(c.workdir, "gopath", "src", mainPkg)
c.mkdirAll(path)
src := fmt.Sprintf(mainSrc, pkg, *flagFunc)
src := fmt.Sprintf(funcMain(), pkg, *flagFunc)
c.writeFile(filepath.Join(path, "main.go"), []byte(src))
return mainPkg
}
Expand Down Expand Up @@ -580,3 +625,46 @@ func main() {
dep.Main(target.%v)
}
`

var mainSrcLibFuzzer = `
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will probably rewrite these to use text/template at some point in the future, since they are pretty large, and it is hard to easily see the formatting directives. Fine for now, though.

package main
import (
"unsafe"
"reflect"
target "%v"
dep "go-fuzz-dep"
)
// #cgo CFLAGS: -Wall -Werror
// #ifdef __linux__
// __attribute__((weak, section("__libfuzzer_extra_counters")))
// #else
// #error Currently only Linux is supported
// #endif
// unsigned char GoFuzzCoverageCounters[65536];
import "C"
//export LLVMFuzzerInitialize
func LLVMFuzzerInitialize(argc uintptr, argv uintptr) int {
dep.Initialize(unsafe.Pointer(&C.GoFuzzCoverageCounters[0]), 65536)
return 0
}
//export LLVMFuzzerTestOneInput
func LLVMFuzzerTestOneInput(data uintptr, size uint64) int {
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!
This improves whole bunch of things -- no copy of data, no construction of Go slice in C code, less C code.

sh := &reflect.SliceHeader{
Data: data,
Len: int(size),
Cap: int(size),
}
input := *(*[]byte)(unsafe.Pointer(sh))
target.%v(input)
return 0
}
func main() {
dvyukov marked this conversation as resolved.
Show resolved Hide resolved
}
`
1 change: 1 addition & 0 deletions go-fuzz-dep/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
// Use of this source code is governed by Apache 2 LICENSE that can be found in the LICENSE file.

// +build gofuzz
// +build !gofuzz_libfuzzer

package gofuzzdep

Expand Down
39 changes: 39 additions & 0 deletions go-fuzz-dep/main_libFuzzer.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
// Copyright 2015 go-fuzz project authors. All rights reserved.
// Use of this source code is governed by Apache 2 LICENSE that can be found in the LICENSE file.

// +build gofuzz
// +build gofuzz_libfuzzer

package gofuzzdep

import (
"unsafe"

. "github.com/dvyukov/go-fuzz/go-fuzz-defs"
)

// Bool is just a bool.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to factor the shared code into a third file that is only protected by the gofuzz build tag.

But again, I'm happy to do that myself as a follow-up after this is merged.

// It is used by code autogenerated by go-fuzz-build
// to avoid compilation errors when a user's code shadows the built-in bool.
type Bool = bool

var (
CoverTab *[CoverSize]byte
sonarRegion []byte
sonarPos uint32
CoverTabTmp [CoverSize]byte
)

func init() {
CoverTab = (*[CoverSize]byte)(unsafe.Pointer(&CoverTabTmp[0]))
}

func Initialize(coverTabPtr unsafe.Pointer, coverTabSize uint64) {
if coverTabSize != CoverSize {
panic("Incorrect cover tab size")
}
CoverTab = (*[CoverSize]byte)(coverTabPtr)
}

func Main(f func([]byte) int) {
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this only in this file, it does not do anything useful. And can confuse future readers as to what's its purpose.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I remove it, I get this:

failed to execute go build: exit status 2
# internal/testlog
/home/jhg/gofuzzpr/go/src/internal/testlog/log.go:69: undefined: gofuzzdep.Main
# errors
/home/jhg/gofuzzpr/go/src/errors/errors.go:20: undefined: gofuzzdep.Main
# math/bits
/home/jhg/gofuzzpr/go/src/math/bits/bits.go:535: undefined: gofuzzdep.Main
/home/jhg/gofuzzpr/go/src/math/bits/bits_tables.go:83: undefined: gofuzzdep.Main
# unicode/utf8
/home/jhg/gofuzzpr/go/src/unicode/utf8/utf8.go:521: undefined: gofuzzdep.Main
# internal/syscall/unix
/home/jhg/gofuzzpr/go/src/internal/syscall/unix/at.go:58: undefined: gofuzzdep.Main
/home/jhg/gofuzzpr/go/src/internal/syscall/unix/at_sysnum_linux.go:13: undefined: gofuzzdep.Main
/home/jhg/gofuzzpr/go/src/internal/syscall/unix/at_sysnum_newfstatat_linux.go:11: undefined: gofuzzdep.Main
/home/jhg/gofuzzpr/go/src/internal/syscall/unix/getrandom_linux.go:46: undefined: gofuzzdep.Main
/home/jhg/gofuzzpr/go/src/internal/syscall/unix/getrandom_linux_amd64.go:9: undefined: gofuzzdep.Main
/home/jhg/gofuzzpr/go/src/internal/syscall/unix/nonblocking.go:17: undefined: gofuzzdep.Main
# unicode
/home/jhg/gofuzzpr/go/src/unicode/casetables.go:20: undefined: gofuzzdep.Main
/home/jhg/gofuzzpr/go/src/unicode/digit.go:13: undefined: gofuzzdep.Main
/home/jhg/gofuzzpr/go/src/unicode/graphic.go:144: undefined: gofuzzdep.Main
/home/jhg/gofuzzpr/go/src/unicode/letter.go:370: undefined: gofuzzdep.Main
/home/jhg/gofuzzpr/go/src/unicode/tables.go:7761: undefined: gofuzzdep.Main

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}