Bazel 1.1 #3249

aherrmann-da · 2019-10-23T08:46:04Z

Update to the latest Bazel version 1.1.0

The Nix expression is taken from bazel: 1.0 -> 1.1 NixOS/nixpkgs#71612.
The Windows dev-env is updated.
rules_scala and rules_nixpkgs are updated for compatibility.
rules_haskell is patched for compatibility. It will be updated in a separate PR.
Some rule attributes and providers were changed.
Some starlark functions no longer accept certain arguments in named form (i.e. as kwargs).
By default Bazel no longer uses a bash test wrapper on Windows, instead uses a C wrapper which calls the test using CreateProcessW. Unfortunately, various tests, including client_server_test, are implemented as bash scripts which cannot be executed with CreateProcessW. Therefore, we disable this new default and use the bash test wrapper. See --incompatible_windows_native_test_wrapper.
Bazel 1.1 fixes an issue with cc_wrapper not working in Cabal packages on Windows for rules_haskell.
Some scala targets were including proto sources in the deps attribute.
This is no longer legal in Bazel 1.1, instead these need to go into the data attribute.

Pull Request Checklist

Read and understand the contribution guidelines
Include appropriate tests
Set a descriptive title and thorough description
Add a reference to the issue this PR will solve, if appropriate
Add a line to the release notes, if appropriate
Normal production system change, include purpose of change in description

NOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with /AzurePipelines run to
trigger the build.

Fixes ``` ERROR: /home/aj/tweag.io/da/da-bazel-1.1/daml-lf/archive/BUILD.bazel:150:1: in deps attribute of scala_test rule //daml-lf/archive:daml_lf_archive_reader_tests_test_suite_src_test_scala_com_digitalasset_daml_lf_archive_DecodeV1Spec.scala: '//daml-lf/archive:daml_lf_1.6_archive_proto_srcs' does not have mandatory providers: 'JavaInfo'. Since this rule was created by the macro 'da_scala_test_suite', the error might have been caused by the macro implementation ```

Bazel 1.1.0 fails on missing hashes.

Modifying sources in-place can cause issues on Windows, where build actions are not sandboxed and changes on sources can affect other build steps.

The bazel-genfiles symlink has been removed since Bazel 1.0. See bazelbuild/bazel#8651

See https://docs.bazel.build/versions/1.1.0/skylark/lib/globals.html#repository_rule

aherrmann-da · 2019-11-11T08:43:33Z

There are some issues with dev_env_tool on Windows, either causing

ERROR: C:/users/vssadministrator/_bazel_vssadministrator/w3d6ug6o/external/nodejs/BUILD.bazel:13:1: @nodejs//:node_bin depends on @nodejs_dev_env//:nodejs_dev_env/node.exe in repository @nodejs_dev_env which failed to fetch. no such package '@nodejs_dev_env//': C:/users/vssadministrator/_bazel_vssadministrator/w3d6ug6o/external/nodejs_dev_env/nodejs_dev_env/node.exe (Permission denied)

ERROR: Analysis of target '//docs:grunt' failed; build aborted: Analysis failed

or

ERROR loading ~/.scoop: The process cannot access the file 'C:\Users\VssAdministrator\.scoop' because it is being used by another process.

However, these are unrelated to this PR and for now worked around with c1e1d1b.

aherrmann-da · 2019-11-11T08:45:28Z

bazel_tools/grpc-bazel-mingw.patch

@@ -20,3 +20,4 @@ index 0349e31bb3..d2e7408dbf 100644
 +#define _WIN32_WINNT 0x0A00

 #include <windows.h>
+


Bazel now uses an internal implementation instead of shelling out to patch. This internal implementation is a bit less lenient than patch and complains about this missing newline.

aherrmann-da · 2019-11-11T08:46:19Z

bazel_tools/haskell-bazel-1.1.0.patch

+
+     # Separate package and target (if present).
+-    package_target = pattern_str[2:].split(":", maxsplit = 2)
+    package_target = pattern_str[2:].split(":", 2)


I'll update rules_haskell in a separate PR, at which point this patch will be unnecessary..

cocreature

Awesome, thank you so much for doing this!

cocreature · 2019-11-11T08:53:45Z

.bazelrc

@@ -67,6 +67,10 @@ build:darwin --host_platform=@rules_haskell//haskell/platforms:darwin_x86_64_nix
 # and GHC's gcc on Windows
 build:windows --crosstool_top=@rules_haskell_ghc_windows_amd64//:cc_toolchain

+# Bazel 1.0 disabled the bash test-runner on Windows. However, some of our
+# test-cases are implemented as bash scripts and rely on the bash test-runner.
+build:windows --noincompatible_windows_native_test_wrapper


Not for this PR but could we work around this by having some wrapper binary that then calls bash internally?

One way to address this is turn things like client_server_test into a macro that generates a script file and then calls sh_test|binary on that. Another is to use cc_toolchain.link|compile to generate a binary that calls the shell script. I think the former is the better approach. I've experimented with both and the latter caused some issues with the test-environment not matching what the test-cases expected.

Sounds good, thanks for the explanation!

* bazel: 0.28.1 --> 1.1.0 * bazel-watcher sha256 * Fix missing line in patch * proto_source_root --> strip_import_prefix See bazelbuild/bazel#7153 for details. * Update rules_nixpkgs Required to avoid errors of the form ``` ERROR: An error occurred during the fetch of repository 'node_nix': parameter 'sep' may not be specified by name, for call to method split(sep, maxsplit = None) of 'string' ``` and ``` ERROR: An error occurred during the fetch of repository 'node_nix': Traceback (most recent call last): File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149 _execute_or_fail(repository_ctx, <3 more arguments>) File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail fail(<1 more arguments>) Cannot build Nix attribute 'nodejs'. Command: [/Users/runner/.nix-profile/bin/nix-build, /private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/node_nix/nix/bazel.nix, "-A", "nodejs", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/nixpkgs/nixpkgs"] Return code: 1 Error output: src/main/tools/process-tools.cc:173: "setitimer": Invalid argument ``` * Update rules_scala * .proto has been removed, use [ProtoInfo] instead See https://docs.bazel.build/versions/1.1.0/be/protocol-buffer.html#proto_library * python3_nix add nix_file attribute To avoid the following error ``` ERROR: /home/aj/tweag.io/da/da-bazel-1.1/BUILD:66:1: //:nix_python3_runtime depends on @python3_nix//:bin/python in repository @python3_nix which failed to fetch. no such package '@python3_nix//': Traceback (most recent call last): File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149 _execute_or_fail(repository_ctx, <3 more arguments>) File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail fail(<1 more arguments>) Cannot build Nix attribute 'python3'. Command: [/home/aj/.nix-profile/bin/nix-build, "-E", "import <nixpkgs> { config = {}; overlays = []; }", "-A", "python3", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs"] Return code: 1 Error output: error: anonymous function at /home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs.nix:3:1 called with unexpected argument 'config', at (string):1:1 ``` * rules_haskell unnamed string.split(_, maxsplit = _) The keyword argument may no longer be named. * string.replace(_, _, maxsplit = _) may not be named * Move proto sources from deps to data Fixes ``` ERROR: /home/aj/tweag.io/da/da-bazel-1.1/daml-lf/archive/BUILD.bazel:150:1: in deps attribute of scala_test rule //daml-lf/archive:daml_lf_archive_reader_tests_test_suite_src_test_scala_com_digitalasset_daml_lf_archive_DecodeV1Spec.scala: '//daml-lf/archive:daml_lf_1.6_archive_proto_srcs' does not have mandatory providers: 'JavaInfo'. Since this rule was created by the macro 'da_scala_test_suite', the error might have been caused by the macro implementation ``` * Define sha256 for haskell_ghc__paths Bazel 1.1.0 fails on missing hashes. * Disable --incompatible_windows_native_test_wrapper * //compiler/daml-extension don't modify sources Modifying sources in-place can cause issues on Windows, where build actions are not sandboxed and changes on sources can affect other build steps. * bazel-genfiles --> bazel-bin The bazel-genfiles symlink has been removed since Bazel 1.0. See bazelbuild/bazel#8651 * Mark dev_env_tool repository rule as configure See https://docs.bazel.build/versions/1.1.0/skylark/lib/globals.html#repository_rule * Move data deps into data attribute * Mark dev_env_tool as local = True * Manually fetch @makensis_dev_env

aherrmann-da force-pushed the bazel-1.1 branch 17 times, most recently from 61217f0 to 7a37811 Compare October 30, 2019 08:49

aherrmann-da force-pushed the bazel-1.1 branch 12 times, most recently from e56d098 to c7d64cb Compare November 6, 2019 10:50

digital-asset deleted a comment from azure-pipelines bot Nov 6, 2019

aherrmann added 8 commits November 8, 2019 10:38

string.replace(_, _, maxsplit = _) may not be named

46eb74e

Define sha256 for haskell_ghc__paths

709a894

Bazel 1.1.0 fails on missing hashes.

Disable --incompatible_windows_native_test_wrapper

5fd186b

//compiler/daml-extension don't modify sources

20131cb

Modifying sources in-place can cause issues on Windows, where build actions are not sandboxed and changes on sources can affect other build steps.

bazel-genfiles --> bazel-bin

3ce076c

The bazel-genfiles symlink has been removed since Bazel 1.0. See bazelbuild/bazel#8651

Mark dev_env_tool repository rule as configure

149002b

See https://docs.bazel.build/versions/1.1.0/skylark/lib/globals.html#repository_rule

Move data deps into data attribute

cbc54a3

aherrmann-da force-pushed the bazel-1.1 branch from b8e89f2 to cbc54a3 Compare November 8, 2019 09:38

aherrmann added 2 commits November 8, 2019 13:55

Mark dev_env_tool as local = True

6640038

Manually fetch @makensis_dev_env

c1e1d1b

aherrmann-da force-pushed the bazel-1.1 branch from 77ebdfc to c1e1d1b Compare November 8, 2019 14:34

aherrmann-da commented Nov 11, 2019

View reviewed changes

aherrmann-da marked this pull request as ready for review November 11, 2019 08:47

aherrmann-da requested review from bame-da, garyverhaegen-da, gerolf-da, hurryabit, leo-da, nemanja-da, S11001001 and stefanobaghino-da as code owners November 11, 2019 08:47

aherrmann-da requested a review from cocreature November 11, 2019 08:47

cocreature approved these changes Nov 11, 2019

View reviewed changes

aherrmann-da merged commit 33e4782 into master Nov 11, 2019

aherrmann-da deleted the bazel-1.1 branch November 11, 2019 09:06

aherrmann-da mentioned this pull request Nov 11, 2019

Mark dev_env_tool as local #3402

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bazel 1.1 #3249

Bazel 1.1 #3249

aherrmann-da commented Oct 23, 2019 •

edited

Loading

aherrmann-da commented Nov 11, 2019

aherrmann-da Nov 11, 2019

aherrmann-da Nov 11, 2019

cocreature left a comment

cocreature Nov 11, 2019

aherrmann-da Nov 11, 2019

cocreature Nov 11, 2019

		@@ -20,3 +20,4 @@ index 0349e31bb3..d2e7408dbf 100644
		+#define _WIN32_WINNT 0x0A00

		#include <windows.h>

Bazel 1.1 #3249

Bazel 1.1 #3249

Conversation

aherrmann-da commented Oct 23, 2019 • edited Loading

Pull Request Checklist

aherrmann-da commented Nov 11, 2019

aherrmann-da Nov 11, 2019

Choose a reason for hiding this comment

aherrmann-da Nov 11, 2019

Choose a reason for hiding this comment

cocreature left a comment

Choose a reason for hiding this comment

cocreature Nov 11, 2019

Choose a reason for hiding this comment

aherrmann-da Nov 11, 2019

Choose a reason for hiding this comment

cocreature Nov 11, 2019

Choose a reason for hiding this comment

aherrmann-da commented Oct 23, 2019 •

edited

Loading