Backports for 1.1.0-rc2 #30607

KristofferC · 2019-01-05T18:08:50Z

Backported PRs:

make inference of _apply match the implementation more closely #30483 - make inference of _apply match the implementation more closely
use type inference world in return_type #30470 - use type inference world in return_type
Use XCode 8.3 for macOS on Travis #30599 - Use XCode 8.3 for macOS on Travis
Fix use counts for mutable struct SROA #30598 - Fix use counts for mutable struct SROA
some latency hacks #30566 - some latency hacks
Fix #29983 #30601 - Fix Uncommenting unused variable assignment (or other various things) gives different results #29983
Force specialization of the SubArray boundserror method. #30666 - Force specialization of the SubArray boundserror method.
Add BB downloading for OpenBLAS #30497 - Add BB downloading for OpenBLAS
Fix bb-install naming conventions, add hashes #30535 - Fix bb-install naming conventions, add hashes

Need manual backport:

Remove duplicate NEWS entry #30560 - Remove duplicate NEWS entry

Non-merged PRs with backport label:

Build: fix output directory when specified with O= #30431 - Build: fix output directory when specified with O=
Include @propagate_inbounds docstring in documentation #30403 - Include @propagate_inbounds docstring in documentation
Fix enumerate documentation #30398 - Fix enumerate documentation
move generate_precompile to Base so that it is bundled in downloads #30367 - move generate_precompile to Base so that it is bundled in downloads
workaround for F_DUPFD_CLOEXEC on Linux kernel <2.6.24 before it existed #30357 - workaround for F_DUPFD_CLOEXEC on Linux kernel <2.6.24 before it existed

…30483) (cherry picked from commit 3e6f607)

ararslan · 2019-01-05T22:54:36Z

I've marked #30599 for backporting, which should fix the macOS CI. @staticfloat, it looks like we'll also need to backport whatever it is you did that fixed AppVeyor.

staticfloat · 2019-01-06T06:04:21Z

So probably 8b189ec then

(cherry picked from commit a5a5882)

This corresponds to macOS 10.12 Sierra. XCode 8 covers El Capitan and Sierra, so if Travis is giving us XCode 8.x for x < 3, we're on El Cap. Homebrew supports only three versions of macOS at a time, which means that El Cap (10.11) is no longer support. This is likely why our Mac builds are trying to build GCC from source; a bottle might not be available. (cherry picked from commit 862fe08)

PR #28478 moved the computation of the use counts before the finish call. to fix #28444. However, the early parts of the finish call fixes up phi node arguments, which fail to get counted if we look at use counts before that fixup is performed. This causes #30594 where the only non-trivial use is on the backedge of the phi and would thus incorrectly fail to get accounted for. Fix that by taking the use count after phi fixup but before dce. (cherry picked from commit f8f2045)

These changes cut some method dependency edges that tend to lead to invalidations when loading packages. (cherry picked from commit 13fc4c3)

SROA was accidentally treating a pending node as old and thus getting the wrong type when querying the predecessor. As a result it thought one of the paths was unreachable causing undefined data to be introduced on that path (generally the `1.0` that happened to already be in register). Fix #29983 (cherry picked from commit da0179c)

I don't have concrete tests for these, but it looks like they all need the `is_old` predicate for what they're doing, so switch those over also while we're at it. (cherry picked from commit 34f7a4a)

KristofferC · 2019-01-09T14:07:23Z

@nanosoldier runbenchmarks("linalg", vs = ":release-1.1")

KristofferC · 2019-01-09T16:42:21Z

@ararslan could you kick nanosoldier?

staticfloat · 2019-01-09T18:51:28Z

The appveyor failure is bizarre; perl is complaining that it can't make changes without a backup here:

julia/deps/blas.mk

Line 96 in 346199d

    
           perl -i -ple 's/^\s*(EXTRALIB\s*\+=\s*-lSystemStubs)\s*$$/# $$1/g' $(dir $<)/Makefile.system

That line hasn't changed in 3 years. Could perl have changed out from underneath us?

staticfloat · 2019-01-09T18:52:10Z

Also why are we even using perl here; we could just turn that into a normal .patch.

ararslan · 2019-01-09T19:39:27Z

@nanosoldier runbenchmarks("linalg", vs=":release-1.1")

vtjnash · 2019-01-09T21:50:42Z

Also why are we even using perl here; we could just turn that into a normal .patch.

We could try, but this bug has been fixed since (at least) v0.1alpha2.5 over 7 years ago, so it would just be an empty file. (OpenMathLib/OpenBLAS@0d76196)

KristofferC · 2019-01-09T22:51:09Z

@nanosoldier runbenchmarks(ALL, vs=":release-1.1")

KristofferC · 2019-01-09T23:00:23Z

I tried to just remove it.

staticfloat · 2019-01-09T23:01:04Z

deps/blas.mk

@@ -92,10 +92,6 @@ endif
 # Do not overwrite the "-j" flag
 OPENBLAS_BUILD_OPTS += MAKE_NB_JOBS=0

-$(BUILDDIR)/$(OPENBLAS_SRC_DIR)/build-configured: $(BUILDDIR)/$(OPENBLAS_SRC_DIR)/source-extracted
-	perl -i -ple 's/^\s*(EXTRALIB\s*\+=\s*-lSystemStubs)\s*$$/# $$1/g' $(dir $<)/Makefile.system


I think you still need to create the build-configured file though; so just remove the perl invocation

Alright, thanks.

nanosoldier · 2019-01-10T05:47:07Z

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @ararslan

KristofferC · 2019-01-10T15:06:45Z

AV seems to build OpenBLAS from scratch now and times out during test on 32 bit but fails on 64 bit with:

cp: cannot stat 'scratch/openblas-fd8d1868a126bb9f12bbc43b36ee30d1ba943fbb/libopenblas64_.dll': No such file or directory
make[1]: *** [/cygdrive/c/projects/julia/deps/blas.mk:112: /cygdrive/c/projects/julia/usr-staging/openblas-fd8d1868a126bb9f12bbc43b36ee30d1ba943fbb.tgz] Error 1
make[1]: Leaving directory '/cygdrive/c/projects/julia/deps'
make: *** [Makefile:57: julia-deps] Error 2

Did I do something bad in the last commit? Help plz.

ararslan · 2019-01-10T20:08:09Z

PkgEval shows no regressions since RC1. Note that it was run prior to the most recent Makefile changes.

staticfloat · 2019-01-10T20:18:38Z

I'm not sure how, but it looks like you only got part of 87c18d8. You need the rest of the changes as well.

KristofferC · 2019-01-10T20:22:40Z

Well, that wasn't marked with backport label and it wasn't the commit in #30607 (comment) so it wasn't backported. I'll add that as well then.

But why do we need so many build backports to this branch? Did something change on AV's end?

staticfloat · 2019-01-10T20:25:01Z

Well, that wasn't marked with backport label

Sorry about that.

But why do we need so many build backports to this branch? Did something change on AV's end?

Yes, the compiler versions changed, but also it's an improvement that reduces compile/test times, so it seemed like a good thing to include.

Improves #29867 by avoiding an invoke. (cherry picked from commit f2d2d6f)

(cherry picked from commit 2a2ba37)

* Auto-detect binarybuilder triplet * Add OpenBLAS BinaryBuilder installation scaffolding Also make it easier to add more BB-cached versions of dependencies in the future * Enable `fixup-libgfortran.sh` to directly ask `$FC` for paths * Tell Appveyor and Travis to use BinaryBuilder OpenBLAS Also allow the build system to auto-guess the triplet (cherry picked from commit 87c18d8)

* Fix `bb-install` naming conventions, add hashes * Set `DEP_LIBS` to include `openblas` on Appveyor * When guessing BB libc, default to `glibc` on Linux * Fix bb-install bash parsing failure Quote to avoid bash freaking out from spurious `)` from compilers that have fancy version strings such as `(Red Hat 7.3.1-5)`, which gets `lastword`'ed down to `7.3.1-5)`. * Add `contrib/refresh_bb_tarballs.sh` to aid in batch-grabbing BB hashes (cherry picked from commit 8b189ec)

KristofferC · 2019-01-11T18:47:27Z

Backported those two but something is still wrong :(

make[1]: Leaving directory '/cygdrive/c/projects/julia'
make[1]: Entering directory '/cygdrive/c/projects/julia'
fatal: cannot change to '/cygdrive/c/projects/julia': No such file or directory

staticfloat · 2019-01-11T18:55:44Z

I have no idea what Appveyor is smoking there. That doesn't make any sense to me, especially as we can see other processes happily going and entering that directory.

KristofferC · 2019-01-11T19:40:48Z

Ok, this happens on master as well. Doesn't stop the tests from running apparently.

Backports for 1.1.0-rc2

don't use abstract_iteration for types with special cases in _apply (#…

41c0c2f

…30483) (cherry picked from commit 3e6f607)

KristofferC added the status:DO NOT MERGE Do not merge this PR! label Jan 5, 2019

JeffBezanson and others added 6 commits January 9, 2019 11:25

use type inference world in return_type (#30470)

879f7c2

(cherry picked from commit a5a5882)

some latency hacks (#30566)

bf0e212

These changes cut some method dependency edges that tend to lead to invalidations when loading packages. (cherry picked from commit 13fc4c3)

Defensively fix patterns similar to #29983

cec7e7b

I don't have concrete tests for these, but it looks like they all need the `is_old` predicate for what they're doing, so switch those over also while we're at it. (cherry picked from commit 34f7a4a)

This comment has been minimized.

Sign in to view

KristofferC mentioned this pull request Jan 9, 2019

remove unneeded patching of Makefile #30672

Merged

staticfloat reviewed Jan 9, 2019

View reviewed changes

KristofferC force-pushed the backport-1.1.0 branch 2 times, most recently from 6ee9441 to 17a5e1e Compare January 9, 2019 23:06

Force specialization of the SubArray boundserror method.

17f2978

Improves #29867 by avoiding an invoke. (cherry picked from commit f2d2d6f)

KristofferC and others added 3 commits January 11, 2019 17:13

remove unneded patching of Makefile

1a2134d

(cherry picked from commit 2a2ba37)

KristofferC force-pushed the backport-1.1.0 branch from 17a5e1e to 05a8951 Compare January 11, 2019 16:15

KristofferC removed the status:DO NOT MERGE Do not merge this PR! label Jan 11, 2019

ararslan merged commit 47038ea into release-1.1 Jan 12, 2019

ararslan deleted the backport-1.1.0 branch January 12, 2019 20:03

vchuravy pushed a commit to JuliaLang/Distributed.jl that referenced this pull request Oct 6, 2023

Merge pull request JuliaLang/julia#30607 from JuliaLang/backport-1.1.0

e168d72

Backports for 1.1.0-rc2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backports for 1.1.0-rc2 #30607

Backports for 1.1.0-rc2 #30607

KristofferC commented Jan 5, 2019 •

edited

Loading

ararslan commented Jan 5, 2019

staticfloat commented Jan 6, 2019

KristofferC commented Jan 9, 2019

KristofferC commented Jan 9, 2019

staticfloat commented Jan 9, 2019

staticfloat commented Jan 9, 2019

ararslan commented Jan 9, 2019

This comment has been minimized.

vtjnash commented Jan 9, 2019 •

edited

Loading

KristofferC commented Jan 9, 2019

KristofferC commented Jan 9, 2019

staticfloat Jan 9, 2019

KristofferC Jan 9, 2019

nanosoldier commented Jan 10, 2019

KristofferC commented Jan 10, 2019

ararslan commented Jan 10, 2019

staticfloat commented Jan 10, 2019

KristofferC commented Jan 10, 2019

staticfloat commented Jan 10, 2019

KristofferC commented Jan 11, 2019

staticfloat commented Jan 11, 2019

KristofferC commented Jan 11, 2019 •

edited

Loading

Backports for 1.1.0-rc2 #30607

Backports for 1.1.0-rc2 #30607

Conversation

KristofferC commented Jan 5, 2019 • edited Loading

ararslan commented Jan 5, 2019

staticfloat commented Jan 6, 2019

KristofferC commented Jan 9, 2019

KristofferC commented Jan 9, 2019

staticfloat commented Jan 9, 2019

staticfloat commented Jan 9, 2019

ararslan commented Jan 9, 2019

This comment has been minimized.

vtjnash commented Jan 9, 2019 • edited Loading

KristofferC commented Jan 9, 2019

KristofferC commented Jan 9, 2019

staticfloat Jan 9, 2019

Choose a reason for hiding this comment

KristofferC Jan 9, 2019

Choose a reason for hiding this comment

nanosoldier commented Jan 10, 2019

KristofferC commented Jan 10, 2019

ararslan commented Jan 10, 2019

staticfloat commented Jan 10, 2019

KristofferC commented Jan 10, 2019

staticfloat commented Jan 10, 2019

KristofferC commented Jan 11, 2019

staticfloat commented Jan 11, 2019

KristofferC commented Jan 11, 2019 • edited Loading

KristofferC commented Jan 5, 2019 •

edited

Loading

vtjnash commented Jan 9, 2019 •

edited

Loading

KristofferC commented Jan 11, 2019 •

edited

Loading