Bazel built at HEAD cannot compile Bazel on OS X

[dhalperi]

Description of the problem / feature request:

Bazel head does not compile on OS X

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

$ git pull && bazel build //src:bazel && cp -f bazel-bin/src/bazel ~/bin
Already up to date.
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 8566f0b5-717f-469c-b5f9-b6aa910b0bba
INFO: Analysed target //src:bazel (217 packages loaded, 10203 targets configured).
INFO: Found 1 target...
ERROR: /Users/dhalperin/git-workspace/bazelbuild/bazel/src/main/cpp/util/BUILD:100:1: Linking of rule '//src/main/cpp/util:numbers' failed (Exit 1) libtool failed: error executing command /usr/bin/libtool rcsD bazel-out/darwin-fastbuild/bin/src/main/cpp/util/libnumbers.a bazel-out/darwin-fastbuild/bin/src/main/cpp/util/_objs/numbers/numbers.pic.o

Use --sandbox_debug to see verbose messages from the sandbox
error: /Library/Developer/CommandLineTools/usr/bin/libtool: no output file specified (specify with -o output)
Usage: /Library/Developer/CommandLineTools/usr/bin/libtool -static [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-sacLT] [-no_warning_for_no_symbols]
Usage: /Library/Developer/CommandLineTools/usr/bin/libtool -dynamic [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-o output] [-install_name name] [-compatibility_version #] [-current_version #] [-seg1addr 0x#] [-segs_read_only_addr 0x#] [-segs_read_write_addr 0x#] [-seg_addr_table <filename>] [-seg_addr_table_filename <file_system_path>] [-all_load] [-noall_load]
Target //src:bazel failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 21.142s, Critical Path: 0.62s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

What operating system are you running Bazel on?

macOS 10.14.3

What's the output of bazel info release?

INFO: Invocation ID: 155ccf1d-a650-4feb-98d8-7764f3dbe57b
development version

It's built and attempted-to-be-rebuilt at

commit 163e61f95153cde12f81003222c07c3697280f24 (HEAD -> master, origin/master, origin/HEAD)
Author: mschaller <mschaller@google.com>
Date:   Mon Feb 11 10:01:33 2019 -0800

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

git@github.com:bazelbuild/bazel.git
163e61f
163e61f

Have you found anything relevant by searching the web?

Some old bugs look vaguely related but also look like fixes were internal.

• Commit a4697fd ("Make C++ archiving use action_configs instead of hardcoded flags") broke cgo builds in MacOS #3184
• error building PCL with cmake_external bazel-contrib/rules_foreign_cc#49
• osx libtool paramater error bazel-contrib/rules_foreign_cc#185

[dhalperi]

Bisecting, was caused by 71bc38f (@scentini)

[dhalperi]

More relevant bugs/links:

• Don't use "libtool" for "ar" on darwin. #5127
• https://github.com/tweag/rules_haskell/pull/223/files

Seems like if #5127 is still "working as intended" then some of the workarounds from rules_haskell or rules_rust need to be incorporated into Bazel.

[dhalperi]

@philwo This also applies to 0.23.0 RC1:

➜  bazel git:(master) ~/Downloads/bazel-0.23.0rc1-darwin-x86_64 build //src:bazel
INFO: Invocation ID: 206665af-9f9d-4c47-ae17-95339fe5cdcb
INFO: Analysed target //src:bazel (217 packages loaded, 10200 targets configured).
INFO: Found 1 target...
ERROR: /Users/dhalperin/git-workspace/bazelbuild/bazel/third_party/grpc/BUILD:1531:1: Linking of rule '//third_party/grpc:grpc_transport_inproc' failed (Exit 1) libtool failed: error executing command /usr/bin/libtool rcsD bazel-out/darwin-fastbuild/bin/third_party/grpc/libgrpc_transport_inproc.a bazel-out/darwin-fastbuild/bin/third_party/grpc/_objs/grpc_transport_inproc/inproc_plugin.pic.o ... (remaining 1 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
error: /Library/Developer/CommandLineTools/usr/bin/libtool: no output file specified (specify with -o output)
Usage: /Library/Developer/CommandLineTools/usr/bin/libtool -static [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-sacLT] [-no_warning_for_no_symbols]
Usage: /Library/Developer/CommandLineTools/usr/bin/libtool -dynamic [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-o output] [-install_name name] [-compatibility_version #] [-current_version #] [-seg1addr 0x#] [-segs_read_only_addr 0x#] [-segs_read_write_addr 0x#] [-seg_addr_table <filename>] [-seg_addr_table_filename <file_system_path>] [-all_load] [-noall_load]
Target //src:bazel failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 16.550s, Critical Path: 10.16s
INFO: 274 processes: 269 remote cache hit, 1 darwin-sandbox, 4 worker.
FAILED: Build did NOT complete successfully

now at

commit 86639a69a0251eb266a10424fd3d1d221a5e476e (HEAD -> master, origin/master, origin/HEAD)
Author: djasper <djasper@google.com>
Date:   Tue Feb 12 12:59:33 2019 -0800

[jin]

Marking this as P0, @philwo / @scentini ptal?

[jin]

FWIW I just built Bazel on HEAD with 0.22 on 10.14.2 and the build succeeded.

[dhalperi]

@jin's comment makes me realize my issue may be unclear.

If B22 is bazel 0.22.0 release, BH is the bazel binary produced by B22 build //src:bazel, then BH build //src:bazel fails at HEAD.

Also, B23RC1 build //src:bazel fails at HEAD.

[hlopko]

Hi Dan,
thanks for the report, we found the problem and the fix is in flight. We're having discussions around the team now. On our CI we use mac workers that have xcode installed (so we can also test objc stuff). It seems that you only have CLT installed, and therefore you use a nonstandard autoconfigured C++ toolchain which contains this bug. Can you tell us what motivated you to only install CLT and not full Xcode?

[philwo]

I tried to repro this, but can't, because my build with Xcode CLT as my system toolchain fails way earlier:

$ sudo xcode-select -s /Library/Developer/CommandLineTools

$ USE_BAZEL_VERSION=0.23.0rc1 bazel clean --expunge
INFO: Invocation ID: 82c242dc-944c-486f-b426-e2bae36f3ad5
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Starting clean.

$ USE_BAZEL_VERSION=0.23.0rc1 bazel build //src:bazel
INFO: Invocation ID: 78143220-8012-4d5e-858a-9b316da60af7
Starting local Bazel server and connecting to it...
ERROR: /private/var/tmp/_bazel_but_philwo/7a01905b4627ca044e5e3f5ad5b14d26/external/local_config_cc/BUILD:55:5: in apple_cc_toolchain rule @local_config_cc//:cc-compiler-ios_armv7: Xcode version must be specified to use an Apple CROSSTOOL. If your Xcode version has changed recently, try: "bazel clean --expunge" to re-run Xcode configuration
ERROR: Analysis of target '//src:bazel' failed; build aborted: Analysis of target '@local_config_cc//:cc-compiler-ios_armv7' failed; build aborted
INFO: Elapsed time: 5.370s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (158 packages loaded, 1912 targets configured)
    currently loading: third_party/aws-sdk-auth-lite ... (3 packages)

I get the same error with Bazel 0.22.0.

When I switch back to full Xcode via sudo xcode-select -r, everything works fine again (including Bazel HEAD being able to build itself).

[philwo]

@dhalperi Can you give us some more information about your system? Maybe the output of these commands helps:

xcodebuild -version
xcode-select -p
clang -v

[jmillikin-stripe]

It seems that you only have CLT installed, and therefore you use a nonstandard autoconfigured C++ toolchain which contains this bug. Can you tell us what motivated you to only install CLT and not full Xcode?

I have a similar setup (as do many coworkers). XCode is heavy and there's no reason to install it for people who don't use it as their editing environment -- if I only want to build C/C++ code, that shouldn't require an IDE.

[dhalperi]

➜  bazel git:(master) xcodebuild -version
xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance
➜  bazel git:(master) xcode-select -p
/Library/Developer/CommandLineTools
➜  bazel git:(master) clang -v
Apple LLVM version 10.0.0 (clang-1000.10.44.4)
Target: x86_64-apple-darwin18.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

I installed Xcode using the "standard" path for CLI homebrew users. brew doctor recommends you install Xcode using xcode-select --install on the CLI, which is all I've ever done.

[dhalperi]

Confirmed fixed by 6ca7763

[philwo]

Thanks for testing! I'll cherry pick this into 0.23.0 then.

[jin]

Closing this as we have a fix, thanks all for looking into this.