Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bazel_4 fails to build on aarch64-darwin #134758

Closed
uri-canva opened this issue Aug 19, 2021 · 19 comments
Closed

bazel_4 fails to build on aarch64-darwin #134758

uri-canva opened this issue Aug 19, 2021 · 19 comments
Labels
0.kind: bug Something is broken 6.topic: darwin Running or building packages on Darwin

Comments

@uri-canva
Copy link
Contributor

Describe the bug

Example build failure: https://hydra.nixos.org/build/150263034/nixlog/1

Steps To Reproduce

Steps to reproduce the behavior:

  1. Build nixpkgs.bazel_4

Expected behavior

Builds correctly.

Notify maintainers

@mboes

Metadata

  • system: "aarch64-darwin"
  • host os: Darwin 20.6.0, macOS 11.5.2
  • multi-user?: yes
  • version: nix-env (Nix) 2.3.15
  • channels(root): "nixpkgs-21.11.pre309670.253aecf69ed"
  • nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixpkgs
@uri-canva uri-canva added the 0.kind: bug Something is broken label Aug 19, 2021
@uri-canva
Copy link
Contributor Author

Couldn't reproduce failure on:

  • system: "x86_64-darwin"
  • host os: Darwin 20.6.0, macOS 10.16
  • multi-user?: yes
  • sandbox: no
  • version: nix-env (Nix) 2.3.12
  • channels(root): "nixpkgs-21.11pre299952.7918dc5148d"
  • channels(uri): "darwin"
  • nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixpkgs

@veprbl veprbl added the 6.topic: darwin Running or building packages on Darwin label Aug 19, 2021
@veprbl
Copy link
Member

veprbl commented Aug 19, 2021

Upstream claims support M1 in v4.1 bazelbuild/bazel#13099 (comment)
Looks like an incompatible argument is supplied to ar

@uri-canva
Copy link
Contributor Author

According to https://github.com/bazelbuild/bazel/releases/tag/4.1.0 the cherry picks requested in that comment were accepted in that version.

@uri-canva
Copy link
Contributor Author

4.2.0 has a different m1 fix: bazelbuild/bazel#13514, might be worth a shot.

@domenkozar
Copy link
Member

#136004 if someone wants to try

@divanorama
Copy link
Contributor

There seems to be one more issue in hydra https://hydra.nixos.org/build/152904307/nixlog/1

checking for references to /build/ in /nix/store/zpwcnd90shsvjrfc8nj88mz79vn0mqim-remote_java_tools_linux...
automatically fixing dependencies for ELF files
searching for dependencies of /nix/store/zpwcnd90shsvjrfc8nj88mz79vn0mqim-remote_java_tools_linux/java_tools/ijar/zipper
  libstdc++.so.6 -> not found!
searching for dependencies of /nix/store/zpwcnd90shsvjrfc8nj88mz79vn0mqim-remote_java_tools_linux/java_tools/ijar/ijar
  libstdc++.so.6 -> not found!
searching for dependencies of /nix/store/zpwcnd90shsvjrfc8nj88mz79vn0mqim-remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local
  libstdc++.so.6 -> not found!
  ld-linux-x86-64.so.2 -> not found!
autoPatchelfHook could not satisfy dependency ld-linux-x86-64.so.2 wanted by /nix/store/zpwcnd90shsvjrfc8nj88mz79vn0mqim-remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local
autoPatchelfHook could not satisfy dependency libstdc++.so.6 wanted by /nix/store/zpwcnd90shsvjrfc8nj88mz79vn0mqim-remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local
Add the missing dependencies to the build inputs or set autoPatchelfIgnoreMissingDeps=true
builder for '/nix/store/vfjl0fc75vqv0nln6jr1zmphi3dkwmvl-remote_java_tools_linux.drv' failed with exit code 1

propagated from 4.1.0 to 4.2.1

@burdiyan
Copy link

Can't use bazel_4 on aarch64-darwin either. Works fine on x86_64-darwin.

Apparently it fails to fetch from the binary cache due to the build failures, and it fails to build locally by throwing this error:

ERROR: /private/tmp/nix-build-bazel-4.2.1.drv-0/bazel_L7PWK63b/out/external/googleapis/BUILD.bazel:201:14: Generating Java (Immutable) proto_library @googleapis//:google_devtools_build_v1_build_status_proto failed: (Segmentation fault): protoc failed: error executing command 
  (cd /private/tmp/nix-build-bazel-4.2.1.drv-0/bazel_L7PWK63b/out/execroot/io_bazel && \
  exec env - \
    PATH=/nix/store/14riyxn9qdjab9mpy6riw2wp97wgg5bl-zip-3.0/bin:/nix/store/nf5jsmymmnz9jxb7s3x0rq40nhf6ffnb-python3-3.9.6/bin:/nix/store/ac212rlx03j6iyi9vzwfhrhx037jc552-unzip-6.0/bin:/nix/store/8y8kcppkv356zk18kkapz2358g9vgnlk-which-2.21/bin:/nix/store/hk8bzk1z63a77f1zydbrdj808hmr9ipa-bash/bin:/nix/store/4lm9am37hnyx3h99w5xzwrsbk91hmh45-cctools-port-949.0.1/bin:/nix/store/ywnnx39dgd515cj4fv9z97mmga8qcbwd-clang-wrapper-11.1.0/bin:/nix/store/wb2g1gnnjwzpcw43pahdmahybzryvhr0-clang-11.1.0/bin:/nix/store/98ggz6lvray5v2kym17yylnnyfz4f4lr-coreutils-8.32/bin:/nix/store/sy8c8580ym2ry7b8ihmggz84a2p16ham-cctools-binutils-darwin-wrapper-949.0.1/bin:/nix/store/7lfjbip2dvhvnhvdj0kx4d63apqlz83v-cctools-binutils-darwin-949.0.1/bin:/nix/store/5sbk39gzizr0cb3ziclhslii6zv71js0-zulu11.48.21-ca-jdk-11.0.11/bin:/nix/store/98ggz6lvray5v2kym17yylnnyfz4f4lr-coreutils-8.32/bin:/nix/store/l1s52qqk0hngyqfjv4938mnc0sbgsjqw-findutils-4.8.0/bin:/nix/store/1ba2zyd9gyal97na97r073g7v6vi87zx-diffutils-3.7/bin:/nix/store/xy0hjszca945iq44z6s29x4n1cc27347-gnused-4.8/bin:/nix/store/fv9ljbpicbqb8h0a40qnz5fmk9s3kb8l-gnugrep-3.6/bin:/nix/store/ijq2s2cn18834ybzsinj1638s2pwp07m-gawk-5.1.0/bin:/nix/store/z4nx7dqji7ziw2hk3apadfc72xjszxww-gnutar-1.34/bin:/nix/store/sqkc7rkjql1r159x903f4r05m2j1sgvn-gzip-1.10/bin:/nix/store/pyr76kmnl8frc71xiyw24q9y7s13jilh-bzip2-1.0.6.0.2-bin/bin:/nix/store/57dwqr6igadm896armq61h96psdni160-gnumake-4.3/bin:/nix/store/cm0ch90wmxl2llj0xjn4rdvmgjn8nr3q-bash-4.4-p23/bin:/nix/store/kki238b0g3l7h5jzndq3l91mpr3b44sc-patch-2.7.6/bin:/nix/store/gx6k0r2zdgfc3mva6fhn7ngdmxyxs5r8-xz-5.2.5-bin/bin:/nix/store/cm0ch90wmxl2llj0xjn4rdvmgjn8nr3q-bash-4.4-p23/bin:/nix/store/98ggz6lvray5v2kym17yylnnyfz4f4lr-coreutils-8.32/bin:/nix/store/l1s52qqk0hngyqfjv4938mnc0sbgsjqw-findutils-4.8.0/bin:/nix/store/ijq2s2cn18834ybzsinj1638s2pwp07m-gawk-5.1.0/bin:/nix/store/fv9ljbpicbqb8h0a40qnz5fmk9s3kb8l-gnugrep-3.6/bin:/nix/store/3zlmz891mnc15flsjrn2wfi97kikicg1-gnutar-1.34/bin:/nix/store/xy0hjszca945iq44z6s29x4n1cc27347-gnused-4.8/bin:/nix/store/sqkc7rkjql1r159x903f4r05m2j1sgvn-gzip-1.10/bin:/nix/store/8y8kcppkv356zk18kkapz2358g9vgnlk-which-2.21/bin:/nix/store/ac212rlx03j6iyi9vzwfhrhx037jc552-unzip-6.0/bin:/nix/store/bnlk9hc1bry2kypj3kc1bz2pxv4h6w88-file-5.40/bin:/nix/store/14riyxn9qdjab9mpy6riw2wp97wgg5bl-zip-3.0/bin:/nix/store/rs534hr95wf5y6wyhdhsg7rd7wdr6zcf-python-2.7.18/bin:/nix/store/nf5jsmymmnz9jxb7s3x0rq40nhf6ffnb-python3-3.9.6/bin \
  bazel-out/darwin_arm64-opt-exec-EDC14992/bin/external/com_google_protobuf/protoc '--proto_path=external/googleapis' '--proto_path=bazel-out/darwin_arm64-opt/bin/external/com_google_protobuf/_virtual_imports/descriptor_proto' '--proto_path=bazel-out/darwin_arm64-opt/bin/external/com_google_protobuf/_virtual_imports/any_proto' '--java_out=bazel-out/darwin_arm64-opt/bin/external/googleapis/google_devtools_build_v1_build_status_proto-speed-src.jar' '-Igoogle/devtools/build/v1/build_status.proto=external/googleapis/google/devtools/build/v1/build_status.proto' '-Igoogle/api/annotations.proto=external/googleapis/google/api/annotations.proto' '-Igoogle/api/http.proto=external/googleapis/google/api/http.proto' '-Igoogle/protobuf/descriptor.proto=bazel-out/darwin_arm64-opt/bin/external/com_google_protobuf/_virtual_imports/descriptor_proto/google/protobuf/descriptor.proto' '-Igoogle/protobuf/any.proto=bazel-out/darwin_arm64-opt/bin/external/com_google_protobuf/_virtual_imports/any_proto/google/protobuf/any.proto' --direct_dependencies google/api/annotations.proto:google/protobuf/any.proto:google/devtools/build/v1/build_status.proto '--direct_dependencies_violation_msg=%s is imported, but @googleapis//:google_devtools_build_v1_build_status_proto doesn'\''t directly depend on a proto_library that '\''srcs'\'' it.' external/googleapis/google/devtools/build/v1/build_status.proto)
Execution platform: //:default_host_platform

@winterqt
Copy link
Member

I personally get this error on aarch64-darwin, but this may be due to having the sandbox enabled?

ERROR: /private/tmp/nix-build-bazel-3.7.2.drv-1/bazel_src/src/BUILD:11:9: Executing genrule //src:install_base_key-file_nojdk failed (Exit 126): bash failed: error executing command
  (cd /private/tmp/nix-build-bazel-3.7.2.drv-1/bazel_87rXB0xe/out/execroot/io_bazel && \
  exec env - \
    PATH=/nix/store/g9wj864wxn9b0gpaq3jlaj6016f82vxn-zip-3.0/bin:/nix/store/6vvgxll8rszf9gmfwrffpljx6ggjz2wf-python3-3.9.6/bin:/nix/store/mz4pp6mzzyjp1qdznzhkwm8qb4algk49-unzip-6.0/bin:/nix/store/jjv1km1xbm7djp5zhilri51bh968n4l4-which-2.21/bin:/nix/store/rmjms2lalf44vcq9xl2qb6wcgras4wqc-bash/bin:/nix/store/90fpiax2n5z2jama4nq6jb74aii0mn47-cctools-port-949.0.1/bin:/nix/store/fyrhb6z8jhq4cn1lyms8mv72ha86pr4a-clang-wrapper-11.1.0/bin:/nix/store/x932wm7dsmycayw4r2mx2sjdy4gdz1hp-clang-11.1.0/bin:/nix/store/czksxcbg7w21f3xb7v18vj58izwfhhwq-coreutils-8.32/bin:/nix/store/c43j0xkpiv1p1vvhvvjqrfj0anya1nz2-cctools-binutils-darwin-wrapper-949.0.1/bin:/nix/store/b8sijxn5c79ca97akmwjc0fw0fbzns4k-cctools-binutils-darwin-949.0.1/bin:/nix/store/1jipfiq5zpkw76lpngkbq8qcnva9cdwn-zulu11.48.21-ca-jdk-11.0.11/bin:/nix/store/czksxcbg7w21f3xb7v18vj58izwfhhwq-coreutils-8.32/bin:/nix/store/xgl93hkxf3dbk1zvsrh3lygsb3qrh78v-findutils-4.8.0/bin:/nix/store/imr49rln4ymkb9xy366h5ksydn3yygwf-diffutils-3.8/bin:/nix/store/i8vzwpizxfjcrvs2xl2vnalhgpzf2q43-gnused-4.8/bin:/nix/store/dbnx8gmcnqha34saw0m0gv81pq71fhds-gnugrep-3.6/bin:/nix/store/xdikb096j4mhhax298mhavadzcf36g8a-gawk-5.1.0/bin:/nix/store/qz7sn4ksivdbvn8k25cr9lb6pl3m34ly-gnutar-1.34/bin:/nix/store/hrshvc98dadw3qddnpyx12r9w620q72b-gzip-1.10/bin:/nix/store/yi3zdan6i5n20vx70a0n564y9ysqdxf9-bzip2-1.0.6.0.2-bin/bin:/nix/store/bbrs6jdw8q7i1i398v5l36pwqsj0z6sf-gnumake-4.3/bin:/nix/store/4ypbwcn0i98vix6b1z0d59d7qgvsxigz-bash-5.1-p8/bin:/nix/store/f6i53mcywvw6xc7wysya6lyi7vp4bkfk-patch-2.7.6/bin:/nix/store/rx2nay69ykn1dzb2dsi9cb7xw5n6d2hr-xz-5.2.5-bin/bin:/nix/store/4ypbwcn0i98vix6b1z0d59d7qgvsxigz-bash-5.1-p8/bin:/nix/store/czksxcbg7w21f3xb7v18vj58izwfhhwq-coreutils-8.32/bin:/nix/store/xgl93hkxf3dbk1zvsrh3lygsb3qrh78v-findutils-4.8.0/bin:/nix/store/xdikb096j4mhhax298mhavadzcf36g8a-gawk-5.1.0/bin:/nix/store/dbnx8gmcnqha34saw0m0gv81pq71fhds-gnugrep-3.6/bin:/nix/store/678x84qa6d2wrnvp8spkyk8qk1kd808j-gnutar-1.34/bin:/nix/store/i8vzwpizxfjcrvs2xl2vnalhgpzf2q43-gnused-4.8/bin:/nix/store/hrshvc98dadw3qddnpyx12r9w620q72b-gzip-1.10/bin:/nix/store/jjv1km1xbm7djp5zhilri51bh968n4l4-which-2.21/bin:/nix/store/mz4pp6mzzyjp1qdznzhkwm8qb4algk49-unzip-6.0/bin:/nix/store/wb7i7kfd6lx2xyxix44ajxqacsdiiab7-file-5.40/bin:/nix/store/g9wj864wxn9b0gpaq3jlaj6016f82vxn-zip-3.0/bin \
  /nix/store/rmjms2lalf44vcq9xl2qb6wcgras4wqc-bash/bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; set -e -o pipefail && md5sum bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar bazel-out/darwin-opt/bin/src/main/java/net/starlark/java/eval/libcpu_profiler.dylib bazel-out/darwin-opt/bin/src/main/cpp/client bazel-out/darwin-opt/bin/src/main/tools/build-runfiles bazel-out/darwin-opt/bin/src/main/tools/process-wrapper bazel-out/darwin-opt/bin/src/main/tools/linux-sandbox bazel-out/darwin-opt/bin/tools/osx/xcode-locator bazel-out/darwin-opt/bin/src/embedded_tools_nojdk.zip | /sbin/md5 | head -c 32 > bazel-out/darwin-opt/bin/src/install_base_key_nojdk')
Execution platform: //:default_host_platform
/nix/store/rmjms2lalf44vcq9xl2qb6wcgras4wqc-bash/bin/bash: line 1: /sbin/md5: Operation not permitted
Target //src:bazel_nojdk failed to build
INFO: Elapsed time: 137.828s, Critical Path: 44.95s
INFO: 2747 processes: 189 internal, 1841 local, 717 worker.
FAILED: Build did NOT complete successfully

ERROR: Could not build Bazel

@cocreature
Copy link
Contributor

I seem to be getting a different error

FAIL: //examples/cpp:hello-success_test (see /private/tmp/nix-build-bazel-4.2.1.drv-0/_bazel_builder/be75a0ea28bc829c1f56b23f6947b302/execroot/io_bazel/bazel-out/darwin_arm64-fastbuild/testlogs/examples/cpp/hello-success_test/test.log)
[19 / 25] 2 actions running
    @bazel_tools//tools/jdk:platformclasspath; 0s processwrapper-sandbox
    Testing //examples/cpp:hello-success_test; 0s processwrapper-sandbox
INFO: From Testing //examples/cpp:hello-success_test:
[19 / 25] 1 / 2 tests, 1 failed; 2 actions running; last test: ...success_test
    @bazel_tools//tools/jdk:platformclasspath; 0s processwrapper-sandbox
    Testing //examples/cpp:hello-success_test; 0s processwrapper-sandbox
==================== Test output for //examples/cpp:hello-success_test:
================================================================================
[19 / 25] 1 / 2 tests, 1 failed; 2 actions running; last test: ...success_test
    @bazel_tools//tools/jdk:platformclasspath; 0s processwrapper-sandbox
    Testing //examples/cpp:hello-success_test; 0s processwrapper-sandbox
[21 / 25] 1 / 2 tests, 1 failed;  1 action; last test: ...p:hello-success_test
    @bazel_tools//tools/jdk:platformclasspath; 0s processwrapper-sandbox
[22 / 25] 1 / 2 tests, 1 failed; 2 actions, 0 running; last test: ...cess_test
    [Prepa] Building .../example/myproject/libhello-lib.jar (1 source file)
    [Prepa] .../java-native/src/main/java/com/example/myproject:hello-lib
[23 / 25] 1 / 2 tests, 1 failed; 2 actions, 1 running; last test: ...cess_test
    .../src/main/java/com/example/myproject:hello-lib; 0s multiplex-worker
    [Prepa] Building .../java/com/example/myproject/hello.jar (1 source file)
[23 / 25] 1 / 2 tests, 1 failed; 2 actions running; last test: ...success_test
    .../src/main/java/com/example/myproject:hello-lib; 0s multiplex-worker
    Building .../myproject/hello.jar (1 source file); 0s multiplex-worker
[25 / 26] 1 / 2 tests, 1 failed;  1 action; last test: ...p:hello-success_test
    [Prepa] Testing //.../src/test/java/com/example/myproject:hello
[25 / 26] 1 / 2 tests, 1 failed;  1 action; last test: ...p:hello-success_test
    Testing //.../java/com/example/myproject:hello; 0s processwrapper-sandbox
INFO: Elapsed time: 9.754s, Critical Path: 2.62s
[26 / 26] 2 / 2 tests, 1 failed; checking cached actions; last test: ...:hello
INFO: 26 processes: 11 internal, 13 processwrapper-sandbox, 2 worker.
[26 / 26] 2 / 2 tests, 1 failed; checking cached actions; last test: ...:hello
INFO: Build completed, 1 test FAILED, 26 total actions
//examples/java-native/src/test/java/com/example/myproject:hello         PASSED in 0.3s
//examples/cpp:hello-success_test                                        FAILED in 0.1s
  /private/tmp/nix-build-bazel-4.2.1.drv-0/_bazel_builder/be75a0ea28bc829c1f56b23f6947b302/execroot/io_bazel/bazel-out/darwin_arm64-fastbuild/testlogs/examples/cpp/hello-success_test/test.log

so it looks like everything built fine but then it fails to run the install tests. The log just seems empty so no idea what is failing.

doInstallCheck gets me a bit further. Not yet at the point where I can say how usable it is.

@liji-canva
Copy link
Contributor

liji-canva commented Nov 10, 2021

I had the same //examples/cpp:hello-success_test error. I found out that the binary built from the cc_test rule returns with error code 137. But I don't have enough knowledge to further debug it.

@winterqt
Copy link
Member

winterqt commented Nov 11, 2021

I'm going to try to tackle this.

Edit: see bazelbuild/bazel#14216, this seems to be an upstream issue not related to Nix.
Is there anyone on an M1 machine running Big Sur that can either get bazel_4 to build, or reproduce the error outlined in this comment? I'm on Monterey so I can't easily test.

Please apply this patch if you get an error about md5sum invocations.

@winterqt
Copy link
Member

cc @NixOS/darwin-maintainers, please see the above comment if you're on aarch64-darwin and are running Big Sur.

@uri-canva
Copy link
Contributor Author

I have Monterey and I can reproduce the protoc error in #134758 (comment).
@liji-canva has Big Sur and cannot reproduce it, it builds correctly for him (minus the tests).

@uri-canva
Copy link
Contributor Author

Hydra doesn't run Monterey yet (it doesn't even run Big Sur), so let's disable doInstallCheck on aarch64-darwin, and people running Monterey will be able to download bazel_4 from the nix binary cache.
I've noticed this issue with some other packages as well: they cannot be built on newer versions of macOS, but downloading them from hydra and running them is all good, I think a lot of those issues are the same @toonn is hitting in the SDK update.

@uri-canva
Copy link
Contributor Author

uri-canva commented Nov 12, 2021

Split out the install check failure out in a separate issue: #145587

@uri-canva
Copy link
Contributor Author

uri-canva commented Nov 12, 2021

Split out the Monterey build failure out in a separate issue: #145590
Note the issue is unrelated to aarch64-darwin, I can reproduce it on x86_64-darwin.

@uri-canva
Copy link
Contributor Author

Sorry for the issue fragmentation, but I believe it will help keep the discussions more focused and make it easier for people experiencing the issues to find them.

I will consider this issue solved once #145588 is merged, and we can continue debugging the other issues separately.

@uri-canva
Copy link
Contributor Author

@winterqt can you open a PR for the md5sum issue in #134758 (comment)?

@winterqt
Copy link
Member

@winterqt can you open a PR for the md5sum issue in #134758 (comment)?

@uri-canva Sure, opened: #145709.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.kind: bug Something is broken 6.topic: darwin Running or building packages on Darwin
Projects
None yet
Development

No branches or pull requests

8 participants