Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add initial support for i386 builds #2347

Merged
merged 12 commits into from
May 13, 2019
14 changes: 12 additions & 2 deletions infra/base-images/base-builder/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,16 @@

FROM gcr.io/oss-fuzz-base/base-clang
MAINTAINER mike.aizatsky@gmail.com
RUN apt-get install -y git subversion jq python3 zip make libunwind8-dev binutils-dev libblocksruntime-dev
RUN apt-get install -y git \
subversion \
jq \
python3 \
zip \
make \
libunwind8-dev \
binutils-dev \
libblocksruntime-dev \
libc6-dev-i386

# Default build flags for various sanitizers.
ENV SANITIZER_FLAGS_address "-fsanitize=address -fsanitize-address-use-after-scope"
Expand All @@ -42,9 +51,10 @@ ENV COVERAGE_FLAGS="-fsanitize=fuzzer-no-link"
# messages which are treated as errors by some projects.
ENV COVERAGE_FLAGS_coverage "-fprofile-instr-generate -fcoverage-mapping -pthread -Wl,--no-as-needed -Wl,-ldl -Wl,-lm -Wno-unused-command-line-argument"

# Default sanitizer and fuzzing engine to use.
# Default sanitizer, fuzzing engine and architecture to use.
ENV SANITIZER="address"
ENV FUZZING_ENGINE="libfuzzer"
ENV ARCHITECTURE="x86_64"

# DEPRECATED - NEW CODE SHOULD NOT USE THIS. OLD CODE SHOULD STOP. Please use
# LIB_FUZZING_ENGINE instead.
Expand Down
4 changes: 4 additions & 0 deletions infra/base-images/base-builder/compile
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ if [ -z "${SANITIZER_FLAGS-}" ]; then
export SANITIZER_FLAGS=${!FLAGS_VAR-}
fi

if [[ $ARCHITECTURE == "i386" ]]; then
export CFLAGS="-m32 $CFLAGS"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you make this a CFLAGS_EXTRA and then add support for that similar to
infra/base-images/base-clang/Dockerfile:ENV CXXFLAGS "$CFLAGS $CXXFLAGS_EXTRA"

Copy link
Contributor Author

@jonathanmetzman jonathanmetzman May 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? as far as I can tell CXXFLAGS_EXTRA is to have "extra" flags to append to CFLAGS to get CXXFLAGS instead of having to repeat the flags. I don't think I understand the benefits having CFLAGS_EXTRA provides.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah my bad, this is flags itself, so appending to CFLAGS directly is fine.

export CXXFLAGS_EXTRA="-L/usr/i386/lib $CXXFLAGS_EXTRA"
fi
if [[ $FUZZING_ENGINE != "none" ]]; then
# compile script might override environment, use . to call it.
. compile_${FUZZING_ENGINE}
Expand Down
16 changes: 15 additions & 1 deletion infra/base-images/base-clang/checkout_build_install_llvm.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
#
################################################################################

LLVM_DEP_PACKAGES="build-essential make cmake ninja-build git subversion python2.7"
LLVM_DEP_PACKAGES="build-essential make cmake ninja-build git subversion python2.7 g++-multilib"
apt-get install -y $LLVM_DEP_PACKAGES

# Checkout
Expand Down Expand Up @@ -99,6 +99,20 @@ ninja
ninja install
rm -rf $WORK/llvm-stage1 $WORK/llvm-stage2

mkdir -p $WORK/i386
cd $WORK/i386
cmake -G "Ninja" \
-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_INSTALL_PREFIX=/usr/i386/ -DLIBCXX_ENABLE_SHARED=OFF \
-DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_FLAGS="-m32" -DCMAKE_CXX_FLAGS="-m32" \
-DLLVM_TARGETS_TO_BUILD="$TARGET_TO_BUILD" \
$SRC/llvm

ninja cxx
ninja install-cxx
rm -rf $WORK/i386

mkdir -p $WORK/msan
cd $WORK/msan

Expand Down
1 change: 1 addition & 0 deletions infra/base-images/base-runner/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ RUN apt-get install -y \
fonts-dejavu \
git \
libblocksruntime0 \
libc6-dev-i386 \
libcap2 \
libunwind8 \
python3 \
Expand Down
11 changes: 10 additions & 1 deletion infra/helper.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ def main():

build_fuzzers_parser = subparsers.add_parser(
'build_fuzzers', help='Build fuzzers for a project.')
_add_architecture_args(build_fuzzers_parser)
_add_engine_args(build_fuzzers_parser)
_add_sanitizer_args(build_fuzzers_parser)
_add_environment_args(build_fuzzers_parser)
Expand All @@ -95,6 +96,7 @@ def main():

check_build_parser = subparsers.add_parser(
'check_build', help='Checks that fuzzers execute without errors.')
_add_architecture_args(check_build_parser)
_add_engine_args(check_build_parser, choices=['libfuzzer', 'afl'])
_add_sanitizer_args(
check_build_parser, choices=['address', 'memory', 'undefined'])
Expand Down Expand Up @@ -149,6 +151,7 @@ def main():
shell_parser = subparsers.add_parser(
'shell', help='Run /bin/bash within the builder container.')
shell_parser.add_argument('project_name', help='name of the project')
_add_architecture_args(shell_parser)
jonathanmetzman marked this conversation as resolved.
Show resolved Hide resolved
_add_engine_args(shell_parser)
_add_sanitizer_args(shell_parser)
_add_environment_args(shell_parser)
Expand Down Expand Up @@ -249,6 +252,11 @@ def _get_work_dir(project_name=''):
return os.path.join(BUILD_DIR, 'work', project_name)


def _add_architecture_args(parser, choices=('x86_64', 'i386')):
"""Add common architecture args."""
parser.add_argument('--architecture', default='x86_64', choices=choices)
jonathanmetzman marked this conversation as resolved.
Show resolved Hide resolved


def _add_engine_args(
parser,
choices=('libfuzzer', 'afl', 'honggfuzz', 'dataflow', 'none')):
Expand Down Expand Up @@ -416,7 +424,8 @@ def build_fuzzers(args):

env = [
'FUZZING_ENGINE=' + args.engine,
'SANITIZER=' + args.sanitizer
'SANITIZER=' + args.sanitizer,
'ARCHITECTURE=' + args.architecture
]
if args.e:
env += args.e
Expand Down
15 changes: 11 additions & 4 deletions infra/travis/travis_build.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@
import subprocess
import yaml


DEFAULT_FUZZING_ENGINES = ['afl', 'libfuzzer']
DEFAULT_SANITIZERS = ['address', 'undefined']

Expand Down Expand Up @@ -54,11 +53,13 @@ def execute_helper_command(helper_command):
subprocess.check_call(command)


def build_fuzzers(project, sanitizer, engine):
def build_fuzzers(project, sanitizer, engine, architecture='x86_64'):
"""Execute helper.py's build_fuzzers command on |project|. Build the fuzzers
with |sanitizer| and |engine|."""
execute_helper_command(
['build_fuzzers', project, '--engine', engine, '--sanitizer', sanitizer])
execute_helper_command([
'build_fuzzers', project, '--engine', engine, '--sanitizer', sanitizer,
'--architecture', architecture
jonathanmetzman marked this conversation as resolved.
Show resolved Hide resolved
])


def check_build(project, sanitizer, engine):
Expand Down Expand Up @@ -97,6 +98,11 @@ def build_project(project):
build_fuzzers(project, sanitizer, 'libfuzzer')
check_build(project, sanitizer, 'libfuzzer')

if 'i386' in project_yaml.get('architectures', []):
# i386 builds always use libFuzzer and ASAN.
build_fuzzers(project, 'address', 'libfuzzer', 'i386')
check_build(project, 'address', 'libfuzzer')


def main():
projects = get_modified_projects()
Expand All @@ -111,5 +117,6 @@ def main():
print('Failed projects:', ' '.join(failed_projects))
exit(1)


if __name__ == '__main__':
main()