Release 3.5 - August 2020 #11885

meisterT · 2020-08-03T07:01:41Z

Tracking bug for the Bazel 3.5 release. Instructions are at https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md

aiuto · 2020-08-06T16:51:28Z

Status update: Branch cut slightly delayed because of power outages after storms in NYC area. Probably will cut on morning of 2020-08-07

davido · 2020-08-10T15:10:04Z

Probably will cut on morning of 2020-08-07

This regression seems to be a release blocker: #11837?

aiuto · 2020-08-12T02:52:09Z

aiuto · 2020-08-12T03:01:59Z

Currently broken and investgating:
flogger: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1594#eda26e49-1c79-4ac6-b9a9-284715b51ac6
google/flogger#179

The two PRs which most likely caused the problems are correctness fixes, so I am not counting this as a blocking change yet.

or-shachar · 2020-08-12T09:29:12Z

Hey hey,
How can we consume rc1? https://releases.bazel.build/3.5.0/rc1/index.html is not available (yet?)

I want to test it against rules_scala and to see that our issue is indeed fixed.

aiuto · 2020-08-12T13:51:56Z

It will be available later today. If you want it earlier, you can clone the repo, check out the release-3.5.0rc1 branch, and build that.

…

On Wed, Aug 12, 2020 at 5:29 AM Or Shachar ***@***.***> wrote: Hey hey, How can we consume rc1? https://releases.bazel.build/3.5.0/rc1/index.html is not available (yet?) I want to test it against rules_scala and to see that our issue is indeed fixed. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#11885 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHEXTAKK4WCED7LJUKDSAJOHRANCNFSM4PTA3RRQ> .

philwo · 2020-08-12T15:12:26Z

FYI, arm64 users: The arm64 build process for this release is still partly manual, so please give me a few minutes after @aiuto pushes rc1, so that I can kick-off the build and upload of the arm64 binaries ;)

aiuto · 2020-08-12T17:30:06Z

It is now available: https://releases.bazel.build/3.5.0/rc1/index.html

davido · 2020-08-12T17:51:31Z

@aiuto

So the fix in #11838 will be included.

The fix from #11838 wasn't included, unfortunately.

Now Bazel-3.5.0rc1 is reporting ca. 100 errors, when trying to build gerrit, that are actually ignored: [1]. You don't see this in Bazel CI, because the errors are actually ignored. Probably other projects are also affected, like JGit and Gitiles.

The only available workaround to silent those errors is to migrate external workspaces from foo-bar to foo_bar. But given that this is a regression, I would rather cherry-pick the PR that is linked to #11838.

Note that all is fine in Bazel 3.4.0 with external workspaces with hyphen char in their names, like foo-bar.

[1] http://paste.openstack.org/show/796786

aiuto · 2020-08-12T19:14:39Z

Oh whoops. Why did I not see that. I can cherrpick it in to rc2

…

On Wed, Aug 12, 2020 at 1:51 PM David Ostrovsky ***@***.***> wrote: @aiuto <https://github.com/aiuto> So the fix in #11838 <#11838> will be included. The fix from #11838 <#11838> wasn't included, unfortunately. Now bazel is reporting ca 100 errors, when trying to build gerrit, that are actually ignored: [1]. The only available workaround to silent those errors is to migrate external workspaces from foo-bar to foo_bar. But given that this is a regression, I would rather cherry-pick the PR that is linked to #11838 <#11838>. [1] http://paste.openstack.org/show/796786 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11885 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHEH3TRF3DWPY3CKIYDSALJDHANCNFSM4PTA3RRQ> .

konste · 2020-08-12T20:41:27Z

Release notes advertise the new feature cc_common.compile support for include_prefix/strip_include_prefix, but it does not seem to be available in 3.5.0RC1
The fix went in just a couple days ago. Is it even going to make it to 3.5.0 release?

aiuto · 2020-08-12T20:51:08Z

I have no answer for why you don't see it. That PR was before the 3.5 baseline, so it should be there.
I would ask @oquenchil about it. If it turns out the release notes got ahead of the functionality, we can fix the release notes more easily then a cherrypick.

konste · 2020-08-12T21:49:30Z

The feature cc_common.compile support for include_prefix/strip_include_prefix does work with 3.5.0RC1
Really sorry for the false alarm! Bazelisk got me confused.

konste · 2020-08-12T22:15:12Z

Seems to be a regression in 3.5.0RC1: #11936

aiuto · 2020-08-13T03:12:12Z

3.5.0rc2 is ready.

davido · 2020-08-13T04:55:12Z

3.5.0rc2 is ready.

Thanks, confirmed that the regression #11838 is fixed in 3.5.0rc2.

or-shachar · 2020-08-13T09:08:10Z

Just checked it on Wix repo - seems like there's a new restriction on external repository names?
we used to have external repositories for docker images with version in their names like busybox_1.25.0 and now we get the error:

10:44:25  Error in workspace: busybox_1.25.0 is not a legal workspace name

I didn't see this in the release notes

meisterT · 2020-08-13T09:09:25Z

@or-shachar see #11936

philwo · 2020-08-13T09:12:42Z

Bazel 3.5.0rc2 Linux arm64 binaries have been successfully built and uploaded to https://releases.bazel.build/3.5.0/rc2/index.html.

davido · 2020-08-13T09:27:37Z

@meisterT @or-shachar

10:44:25 Error in workspace: busybox_1.25.0 is not a legal workspace name

If it is your primary WORKSPASE file, then I think this is rather the same regression as in #11837. I only added hyphen char in my fix, because Gerrit project doesn't use dot char in workspace repository names.

See also this #11838 (comment) where @Gromba asked to allow dot char in repository names as well.

or-shachar · 2020-08-13T10:04:46Z

@davido @meisterT - it's not the primary workspace name but a generated workspace from the rule container_pull. We had dots in the external repository name which worked fine until this version

aiuto · 2020-08-13T13:51:06Z

I'll do some hunting to figure out why we added the restriction in the first place. That will help me decide if we do a patch in the minor releases until the next major and break then, or revert totally.

aiuto · 2020-08-13T15:35:06Z

I have a change in the pipeline to allow dot in workspace names. That will roll into RC3

laurentlb · 2020-08-13T15:43:36Z

The issue is discussed on #11837

aiuto · 2020-08-13T18:32:34Z

We can resolve the prelude processing problem #11940 by picking a baseline from Aug. 7. 889bc0b

Now the question is if the new error reporting is a blocker or not.

are there reasonable workarounds?
is there a short fix forward to suppress the better error reporting?

And also, do we revert the change allowing hyphens in names first. Given today's discussions, that is probably called for first.
I'll think through all this by Friday EOD and come up with a plan.

aiuto · 2020-08-13T20:49:27Z

Background on the flogger issue: bazelbuild/continuous-integration#1006
The proposal was to provide the needed Android SDK in the CI environment.

vbatts · 2020-09-09T16:12:32Z

I'm late to the party.
It looks like c++ error on g++ 10.2.1 (on fedora 32)?
This is from a local build, and i've kicked off a copr build for all platforms: https://copr.fedorainfracloud.org/coprs/build/1654639

ERROR: /var/tmp/bazel_QlyAxJ5T/out/external/upb/BUILD:57:11: C++ compilation of rule '@upb//:upb' failed (Exit 1): gcc failed: error executing command
  (cd /var/tmp/bazel_QlyAxJ5T/out/execroot/io_bazel && \
  exec env - \
    PATH=/home/vbatts/workspace/src/github.com/vbatts/copr-build-bazel/bazel-3.5.0/bin-hack:/home/vbatts/workspace/bin:/home/vbatts/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/sbin \
    PWD=/proc/self/cwd \
  /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections -MD -MF bazel-out/k8-opt/bin/external/upb/_objs/upb/upb.d '-frandom-seed=bazel-out/k8-opt/bin/external/upb/_objs/upb/upb.o' -iquote external/upb -iquote bazel-out/k8-opt/bin/external/upb -Werror -Wno-long-long -pedantic -Wstrict-prototypes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/upb/upb/upb.c -o bazel-out/k8-opt/bin/external/upb/_objs/upb/upb.o)
Execution platform: //:default_host_platform
In file included from /usr/include/string.h:495,
                 from external/upb/upb/upb.h:16,
                 from external/upb/upb/upb.c:2:
In function 'strncpy',
    inlined from 'upb_status_seterrmsg' at external/upb/upb/upb.c:40:3:
/usr/include/bits/string_fortified.h:106:10: error: '__builtin_strncpy' specified bound 127 equals destination size [-Werror=stringop-truncation]
  106 |   return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest));
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
Target //src:bazel_nojdk failed to build
INFO: Elapsed time: 20.600s, Critical Path: 2.91s
INFO: 60 processes: 60 local.
FAILED: Build did NOT complete successfully

$ g++ --version
g++ (GCC) 10.2.1 20200723 (Red Hat 10.2.1-1)
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

aiuto · 2020-09-09T16:20:18Z

This is for the Fedora bless package of Bazel 3.5, so it is mostly about building Bazel. Right?
Even so, does the 3.5 binary (un-repackaged) work with user projects using C++? Or, are the rule broken as well with g++ 10.2.1

aiuto · 2020-09-09T16:34:44Z

It looks like it fails for all platforms in @upb, which gRPC uses. My hunch is that this is related to a 3.4 problem with gRPC that caused some rollback/forward and a 3.4.1 release.
#11792
c3ea5a1
fa7299c

@meteorcloudy Do you have any ideas?

vbatts · 2020-09-09T20:54:05Z

This is for the Fedora bless package of Bazel 3.5, so it is mostly about building Bazel. Right?
Even so, does the 3.5 binary (un-repackaged) work with user projects using C++? Or, are the rule broken as well with g++ 10.2.1

this is in building bazel itself.
https://github.com/vbatts/copr-build-bazel/tree/v3.5.0-1
run: make rpm from a fedora 32 host.

meteorcloudy · 2020-09-10T08:19:39Z

Looks like it's the same issue as #12056

upb has fixed the problem, we should update the upb version in Bazel.

meteorcloudy · 2020-09-10T08:33:43Z

PR sent: #12077

aiuto · 2020-09-11T16:12:10Z

@meteorcloudy Is it time to try a 3.5.1 with the cherrypick of b3ac8f6
or is there more to do?

aiuto · 2020-09-11T16:13:45Z

Actually, @vbatts, perhaps you are in the best situation to test, as you have the right compiler environment.
If you patch that commit into the 3.5.0 sources does your build then work?

Ref: bazelbuild/bazel#11885 (comment) Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>

vbatts · 2020-09-11T18:01:41Z

tested b3ac8f6, and it seems to rely on other files than are included in 3.5.0: ``` File "/home/vbatts/workspace/src/github.com/vbatts/copr-build-bazel/bazel-3.5.0/tools/build_defs/repo/http.bzl", line 121, column 10, in _http_archive_impl patch(ctx) File "/home/vbatts/workspace/src/github.com/vbatts/copr-build-bazel/bazel-3.5.0/tools/build_defs/repo/utils.bzl", line 135, column 22, in patch ctx.patch(patchfile, strip) Error in patch: Not a regular file: /home/vbatts/workspace/src/github.com/vbatts/copr-build-bazel/bazel-3.5.0/third_party/grpc/upb_gcc10_fix.patch ERROR: no such package '@upb//bazel': Not a regular file: /home/vbatts/workspace/src/github.com/vbatts/copr-build-bazel/bazel-3.5.0/third_party/grpc/upb_gcc10_fix.patch ```

vbatts · 2020-09-11T20:39:29Z

applied 26cbf77 also
and local build is making it further, so i've kicked off https://copr.fedorainfracloud.org/coprs/build/1658045

vbatts · 2020-09-12T00:09:50Z

Those two commits did the trick.
(The two failures are the couple of aarch64 that have consistently been failing)

aiuto · 2020-09-12T02:21:42Z

Thanks.

I started a 3.5.1 run with those two cherrypicks + correctness bug fix from b9bb102.

scripts/release/release.sh create --force_rc=1 3.5.1 \
	889bc0b523b47eeb38a72bf9bb6858ee525a7c7e \
	a7a0d48fbeb059ee60e77580e5d05baeefdd5699 \
	b37c51c7085f0aefe04034dd451acb847605ddb5 \
	f6ad35fcde93f92c591778ed7db38d167f5bbc03 \
	39bc97eab295bddb35b38bfc4a2ff3d2b15d034e \
	b9706675a7abf6ceebb250f0b3dfa4087a0c35f6 \
	e574d558da17cfd0f818e7a937a07926aa270069 \
	9993785fa0c4fa4172aa31d306f3abea76833abf \
	b3ac8f60973ba60d578ae6a653cdd993a2d206d7 \
	26cbf776e0cdd8fbb9a40aca4c1be6f5313b6eb4 \
	b9bb10252a004047f9a997165aa1ab7d6c6fa2e7

Alas, it seems to fail on an unrelated issue.

src/tools/android/java/com/android/tools/r8/CompatDxSupport.java:16: error: cannot find symbol
--
  | import com.android.tools.r8.utils.AndroidApp;

https://buildkite.com/bazel/bazel-bazel/builds?branch=release-3.5.1rc1.

I'll have to dig into that.

philwo · 2020-09-12T06:51:23Z

@aiuto This looks like an incompatibility between Bazel 3.5.x and a too new Android SDK. IIRC we added a newer version of the Android build-tools on the CI machines in the meantime due to a request from the Android folks. Because Bazel's WORKSPACE build does not pin the tools to a fixed version, Bazel 3.5.x now picks up that latest version, but is not yet compatible with it.

aiuto · 2020-09-12T15:30:44Z

On Sat, Sep 12, 2020 at 2:51 AM Philipp Wollermann ***@***.***> wrote: @aiuto <https://github.com/aiuto> This looks like an incompatibility between Bazel 3.5.x and a too new Android SDK.

It does look like that.

IIRC we added a newer version of the Android build-tools on the CI machines in the meantime due to a request from the Android folks. Because Bazel's WORKSPACE build does not pin the tools to a fixed version, Bazel 3.5.x now picks up that latest version, but is not yet compatible with it.

Wouldn't Bazel at head fail in CI if we did not have a corresponding change at head to account for the new SDK? Last night I started looking through all the commits between Aug 7 and now but did not find evidence of that. Perhaps it was just too long a day. Are there notes about how we put the SDK on the CI machines? Those would be handy for setting up personal machines to do full testing locally.

aiuto · 2020-09-14T19:12:33Z

@philwo The obvious question is you say "we added a newer version. If you really mean added, rather than replaced, this should only require pinning the version in the WORKSPACE.

philwo · 2020-09-14T19:21:36Z

@aiuto Yes, we added Android build-tools 30.0.1 and removed unused versions 28.0.3 and 29.0.0: bazelbuild/continuous-integration@050153d. This means, we have build-tools 28.0.2, 29.0.2, 29.0.3 and 30.0.1 available on the CI machines.

Before this change, Bazel would have used build-tools 29.0.3, after this change it would use 30.0.1.

Bazel at HEAD and Bazel 3.6 are compatible with build-tools 30.0.1, but Bazel 3.5 is not yet compatible.

I believe the commits that added the compatibility are these:

You can check Google bug b/158484558 for some context.

I think as a way forward we have two options:

Pin Android build-tools to 29.0.3 in the WORKSPACE of Bazel 3.5.x. This is the lowest risk, but a bit annoying as we do not have this as a commit we can cherry-pick, so we would somehow have to get it into master first and then cherry-pick it and then change master again to be pinned to build-tools 30.0.1.
Or we can cherry-pick the above commits.

aiuto · 2020-09-25T13:58:46Z

I am trying the cherrypicks. I'll know if this works in about 50 minutes.

scripts/release/release.sh create --force_rc=1 \
	3.5.1 889bc0b523b47eeb38a72bf9bb6858ee525a7c7e \
	a7a0d48fbeb059ee60e77580e5d05baeefdd5699 \
	b37c51c7085f0aefe04034dd451acb847605ddb5 \
	f6ad35fcde93f92c591778ed7db38d167f5bbc03 \
	39bc97eab295bddb35b38bfc4a2ff3d2b15d034e \
	b9706675a7abf6ceebb250f0b3dfa4087a0c35f6 \
	e574d558da17cfd0f818e7a937a07926aa270069 \
	9993785fa0c4fa4172aa31d306f3abea76833abf \
	b3ac8f60973ba60d578ae6a653cdd993a2d206d7 \
	26cbf776e0cdd8fbb9a40aca4c1be6f5313b6eb4 \
	b9bb10252a004047f9a997165aa1ab7d6c6fa2e7 \
	6b591a7e20f05224898b31ad5b03ba514eff6118 \
	7a11752a8ae7689d2bd482e23d466cb44a3261a1

aiuto · 2020-09-25T15:50:12Z

Pushed release: 3.5.1rc1

@vbatts @petemounce @excitoon : Could you please test that this can be repackaged in your respective systems.

aiuto · 2020-09-29T16:38:10Z

Since yesterday was a holiday, I will let this age one more business day before dropping the "rc1" and upgrading to 3.5.1

philwo · 2020-09-30T21:35:23Z

Can we release Bazel 3.5.1? Due to a bug in Bazelisk (bazelbuild/bazelisk#170), certain builds around the world are failing until we do so.

aiuto · 2020-09-30T22:12:49Z

I will tonight. I was hoping to get a thumbs up from the downstream packagers, but I don't want to keep you blocked.

…

On Wed, Sep 30, 2020, 5:35 PM Philipp Wollermann ***@***.***> wrote: Can we release Bazel 3.5.1? Due to a bug in Bazelisk ( bazelbuild/bazelisk#170 <bazelbuild/bazelisk#170>), certain builds around the world are failing until we do so. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11885 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHDX2IFUPEH53574L5TSIOQCXANCNFSM4PTA3RRQ> .

aiuto · 2020-10-01T17:45:06Z

I just pushed release: 3.5.1

https://github.com/bazelbuild/bazel/releases/tag/3.5.1

@vbatts @petemounce @excitoon : Could you please repackage this for your respective systems.

vbatts · 2020-10-01T22:00:43Z

Fedora and Centos builds are kicked off: https://copr.fedorainfracloud.org/coprs/build/1691954

vbatts · 2020-10-02T14:10:50Z

I forgot to remove the cherry-picked patches, so that build failed.
Kicked off a new build: https://copr.fedorainfracloud.org/coprs/build/1692951

philwo · 2020-10-06T14:50:57Z

Bazel 3.5.1 arm64 binaries uploaded to GitHub and https://releases.bazel.build/3.5.1/release/.

meisterT assigned aiuto Aug 3, 2020

meisterT added the release label Aug 3, 2020

meisterT pinned this issue Aug 3, 2020

meisterT mentioned this issue Aug 3, 2020

Release 3.4 - July 2020 #11687

Closed

10 tasks

aiuto closed this as completed Sep 9, 2020

aiuto reopened this Sep 9, 2020

vbatts added a commit to vbatts/copr-build-bazel that referenced this issue Sep 11, 2020

Testing upb fix from upstream

8cd275f

Ref: bazelbuild/bazel#11885 (comment) Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>

sventiffe closed this as completed Oct 7, 2020

laurentlb unpinned this issue Oct 9, 2020

Release 3.5 - August 2020 #11885

Release 3.5 - August 2020 #11885

Comments

meisterT commented Aug 3, 2020

aiuto commented Aug 6, 2020

davido commented Aug 10, 2020

aiuto commented Aug 12, 2020 • edited Loading

Status of Bazel 3.5

aiuto commented Aug 12, 2020 • edited Loading

or-shachar commented Aug 12, 2020

aiuto commented Aug 12, 2020 via email

philwo commented Aug 12, 2020

aiuto commented Aug 12, 2020

davido commented Aug 12, 2020 • edited Loading

aiuto commented Aug 12, 2020 via email

konste commented Aug 12, 2020

aiuto commented Aug 12, 2020

konste commented Aug 12, 2020

konste commented Aug 12, 2020

aiuto commented Aug 13, 2020

davido commented Aug 13, 2020

or-shachar commented Aug 13, 2020

meisterT commented Aug 13, 2020

philwo commented Aug 13, 2020 • edited Loading

davido commented Aug 13, 2020 • edited Loading

or-shachar commented Aug 13, 2020

aiuto commented Aug 13, 2020

aiuto commented Aug 13, 2020

laurentlb commented Aug 13, 2020

aiuto commented Aug 13, 2020

aiuto commented Aug 13, 2020

vbatts commented Sep 9, 2020 • edited Loading

aiuto commented Sep 9, 2020

aiuto commented Sep 9, 2020 • edited Loading

vbatts commented Sep 9, 2020

meteorcloudy commented Sep 10, 2020

meteorcloudy commented Sep 10, 2020

aiuto commented Sep 11, 2020

aiuto commented Sep 11, 2020

vbatts commented Sep 11, 2020 via email

vbatts commented Sep 11, 2020

vbatts commented Sep 12, 2020

aiuto commented Sep 12, 2020

philwo commented Sep 12, 2020

aiuto commented Sep 12, 2020 via email

aiuto commented Sep 14, 2020

philwo commented Sep 14, 2020 • edited Loading

aiuto commented Sep 25, 2020

aiuto commented Sep 25, 2020

aiuto commented Sep 29, 2020

philwo commented Sep 30, 2020

aiuto commented Sep 30, 2020 via email

aiuto commented Oct 1, 2020

vbatts commented Oct 1, 2020 via email

vbatts commented Oct 2, 2020

philwo commented Oct 6, 2020

aiuto commented Aug 12, 2020 •

edited

Loading

aiuto commented Aug 12, 2020 •

edited

Loading

davido commented Aug 12, 2020 •

edited

Loading

philwo commented Aug 13, 2020 •

edited

Loading

davido commented Aug 13, 2020 •

edited

Loading

vbatts commented Sep 9, 2020 •

edited

Loading

aiuto commented Sep 9, 2020 •

edited

Loading

philwo commented Sep 14, 2020 •

edited

Loading